SDK ignores "Content-Length" header of a S3 PutObject request with the body backed by an InputStream #2908

tatsuya6502 · 2021-12-10T09:23:34Z

Describe the bug

NOTE: This issue is similar to #460, which was fixed in 2.0.0-preview-11. The only difference is that in #460, the body was created from a String, and in this one, the body is created from an InputStream.

If I create a S3 PutObject request using an InputStream and I incorrectly specify the size of the body LESS than the size of the payload, SDK will actually send the entire payload.

private static final int OBJECT_SIZE = 512; // bytes

public static void main(...) throws ... {
    S3Client client = createS3Client();

    PutObjectRequest req = PutObjectRequest.builder().bucket(BUCKET_NAME).key(KEY).build();

    // Create the payload whose size is 512 bytes.
    InputStream is = createInputStreamForBody(OBJECT_SIZE);
    // Create the request body with the InputStream, but with a wrong `contentLength` 256.
    RequestBody body = RequestBody.fromInputStream(is, OBJECT_SIZE / 2);
    client.putObject(req, body);
}

/**
 * Returns an InputStream backed by a byte array of `size` filled with random data.
 */
private InputStream createInputStreamForBody(int size) {
    Random rnd = new Random();
    byte[] bytes = new byte[size];
    rnd.nextBytes(bytes);
    return new ByteArrayInputStream(bytes);
}

The SDK sends the header and the first chunk of the body:

PUT /key1 HTTP/1.1
Host: <omitted>.s3.ap-northeast-1.amazonaws.com
amz-sdk-invocation-id: <omitted>
amz-sdk-retry: 0/0/500
Authorization: AWS4-HMAC-SHA256 Credential=<omitted>/20211210/ap-northeast-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=<omitted>
Content-Type: application/octet-stream
Expect: 100-continue
User-Agent: aws-sdk-java/2.10.54 Linux/5.10.81 OpenJDK_64-Bit_Server_VM/17.0.1+12-nixos Java/17.0.1 vendor/N_A io/sync http/Apache
x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD
X-Amz-Date: 20211210T073003Z
x-amz-decoded-content-length: 256
Content-Length: 430
Connection: Keep-Alive

200;chunk-signature=<omitted>
.....

(200 is the size of the chunk in hexadecimal. 0x200 = 512 bytes)

And Amazon S3 returns the following error response:

HTTP/1.1 400 Bad Request
x-amz-request-id: <omitted>
x-amz-id-2: <omitted>
Content-Type: application/xml
Transfer-Encoding: chunked
Date: Fri, 10 Dec 2021 07:30:02 GMT
Server: AmazonS3
Connection: close

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>IncompleteBody</Code><Message>The request body terminated unexpectedly</Message>
<RequestId><omitted></RequestId><HostId><omitted></HostId></Error>

Expected behavior

I would expect that the SDK to stop reading from the InputStream when it has read the contentLength bytes.

Java API: software.amazon.awssdk.core.sync.RequestBody#fromInputStream()

public static RequestBody fromInputStream(InputStream inputStream, long contentLength)

Parameters:

inputStream - Input stream to send to the service. The stream will not be closed by the SDK.
contentLength - Content length of data in input stream.

Current behavior

The SDK does not respect the contentLength, and tries to send all bytes read from the InputStream. This causes Amazon S3 to return 400 Bad Request error.

Steps to Reproduce

Create the request body using RequestBody.fromInputStream(InputStream inputStream, long contentLength)
- But make contentLength smaller than the actual size of the payload inputStream.
Send a S3 PutObject request with the request body.

(I will provide a full Java source code later)

Possible Solution

Update the SDK so that it will stop reading from the InputStream when it has read the contentLength bytes.

Context

For example, I may want to send and store only first portion of a local file to Amazon S3, by setting the contentLength smaller than the actual file size.

AWS Java SDK version used

2.10.54 and 2.17.98

JDK version used

openjdk version "11.0.12" 2021-07-20 64-Bit Server VM

Operating System and version

Linux x86_64 (NixOS 22.05pre Quokka)

The text was updated successfully, but these errors were encountered:

aaoswal · 2022-02-03T19:25:40Z

Hi @tatsuya6502,
Could you please provide the following:

Full sample code for reproduction of the issue
Full stack trace of the error

Thanks,
Anish

github-actions · 2022-02-08T21:05:28Z

It looks like this issue has not been active for more than five days. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please add a comment to prevent automatic closure, or if the issue is already closed please feel free to reopen it.

tatsuya6502 · 2022-02-09T04:39:34Z

Hi @aaoswal,

Thank you for looking into this issue.

Could you please provide the following:

Full sample code for reproduction of the issue

Full stack trace of the error

Full sample code for reproduction of the issue

I uploaded a full sample here: https://github.com/tatsuya6502/aws-sdk-java-v2-gh2908

You can run it by the followings commands:

$ git clone https://github.com/tatsuya6502/aws-sdk-java-v2-gh2908.git
$ cd aws-sdk-java-v2-gh2908

$ ./gradlew run --args='--bucket mybucket -H'

Notes:

Replace mybucket with a real bucket name.
The bucket will be created in Tokyo region ap-northeast-1. You can change this by editing the Java source file below.

All Java source code is in this file:

https://github.com/tatsuya6502/aws-sdk-java-v2-gh2908/blob/main/app/src/main/java/chunk/tester/App.java

Full stack trace of the error

$ ./gradlew run --args='-b bn2908-0204a -H'

...

PutObject: Sending a request. (key: key1, object size: 512, content length: 256)

2022-02-09 12:33:15 [main] DEBUG software.amazon.awssdk.request:85 - Sending Request: DefaultSdkHttpFullRequest(httpMethod=PUT, protocol=http, host=bn2908-0204a.s3.ap-northeast-1.amazonaws.com, port=80, encodedPath=/key1, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, Expect, User-Agent], queryParameters=[])
2022-02-09 12:33:15 [main] DEBUG org.apache.http.wire:73 - http-outgoing-0 >> "PUT /key1 HTTP/1.1[\r][\n]"
2022-02-09 12:33:15 [main] DEBUG org.apache.http.wire:73 - http-outgoing-0 >> "Host: bn2908-0204a.s3.ap-northeast-1.amazonaws.com[\r][\n]"
2022-02-09 12:33:15 [main] DEBUG org.apache.http.wire:73 - http-outgoing-0 >> "amz-sdk-invocation-id: c0c0953c-b6cc-8f68-334c-84f943f0aeeb[\r][\n]"
2022-02-09 12:33:15 [main] DEBUG org.apache.http.wire:73 - http-outgoing-0 >> "amz-sdk-request: attempt=1; max=4[\r][\n]"
2022-02-09 12:33:15 [main] DEBUG org.apache.http.wire:73 - http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=AKI*****/20220209/ap-northeast-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=418e*****e8cf[\r][\n]"
2022-02-09 12:33:15 [main] DEBUG org.apache.http.wire:73 - http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
2022-02-09 12:33:15 [main] DEBUG org.apache.http.wire:73 - http-outgoing-0 >> "Expect: 100-continue[\r][\n]"
2022-02-09 12:33:15 [main] DEBUG org.apache.http.wire:73 - http-outgoing-0 >> "User-Agent: aws-sdk-java/2.17.126 Mac_OS_X/11.6.3 OpenJDK_64-Bit_Server_VM/11.0.12+0 Java/11.0.12 vendor/Homebrew io/sync http/Apache cfg/retry-mode/legacy[\r][\n]"
2022-02-09 12:33:15 [main] DEBUG org.apache.http.wire:73 - http-outgoing-0 >> "x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD[\r][\n]"
2022-02-09 12:33:15 [main] DEBUG org.apache.http.wire:73 - http-outgoing-0 >> "X-Amz-Date: 20220209T043315Z[\r][\n]"
2022-02-09 12:33:15 [main] DEBUG org.apache.http.wire:73 - http-outgoing-0 >> "x-amz-decoded-content-length: 256[\r][\n]"
2022-02-09 12:33:15 [main] DEBUG org.apache.http.wire:73 - http-outgoing-0 >> "Content-Length: 430[\r][\n]"
2022-02-09 12:33:15 [main] DEBUG org.apache.http.wire:73 - http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
2022-02-09 12:33:15 [main] DEBUG org.apache.http.wire:73 - http-outgoing-0 >> "[\r][\n]"
2022-02-09 12:33:15 [main] DEBUG org.apache.http.wire:73 - http-outgoing-0 << "HTTP/1.1 100 Continue[\r][\n]"
2022-02-09 12:33:15 [main] DEBUG org.apache.http.wire:73 - http-outgoing-0 << "[\r][\n]"
2022-02-09 12:33:15 [main] DEBUG org.apache.http.wire:73 - http-outgoing-0 >> "200;chunk-signature=469b918c28d4a24ca832a5ead443ccce19ca05027cd3d6eaf0f01b0602373d46[\r][\n]"
2022-02-09 12:33:15 [main] DEBUG org.apache.http.wire:73 - http-outgoing-0 >> "[0xcf][0x93][0xdb]1[0xd1]b[0x0][0xb6][0xf9][0xb3][0x82]\[0xb1][0x88]K[0xad]Vg[0xcb][0xf6]n[0xff]#[0xd4][0x95]/[0x15][0xb1]R[0xf0]d[0x19][0xd7]dT[0x88][0x1b][0xd6]I[0x3][0xeb]z[0xcf]7+[0xbc]u[0xb8]!r[0xb0]84[0xb7][0xef][0xe1]'w[0xb0][0xc7][0xc8]z[0xd3][0xd6][0x9e]'[0xb7]hA[0x14][0xe2][0x9f];M7[0xce][0xc7][0x9a]W[0xae][0xb8]7C[0x1d][0x8e][0xdd]|[0xbd][0xe0][0xbd][0x84][0xa1][0x84][0x14]"[0x88]xh|[0xcd][0xfa]$[0xa4][0xb2][0xda][0xed][0xf2][0xed][0x16]{p[0xe]p[0x3][0x95][0xad][0xf][0x9a]#[0xd2]2[0xa7][0x9][0xa8] [0xe1][0xdb][0xdf][0x13])S3[0x81]qy[0xa2][0xd5][0x1e][0x1e]y5[0xb5][0x2]k[0xb2][0xf5][0xdd][0x98]@[0xe1][0xca]n[0xa6][\n]"
2022-02-09 12:33:15 [main] DEBUG org.apache.http.wire:73 - http-outgoing-0 >> "hJ'[0xd2][0xe0]Rh[0xa6]*$Z[0xb][0x10]H)o[0xd7]01[0xb6][0x8][0xb7][0xb4]S0[0x89][0x81][0xa6][\n]"
2022-02-09 12:33:15 [main] DEBUG org.apache.http.wire:73 - http-outgoing-0 >> "T[0xc3]:![0x19]gX[0xc6][0x3][0xcf]T[0xb4]|[0x5]2ZwH[0x3][0xad]Hp[0xa3][0x85][0xdb]g[0x13][0xd8][0xeb][0xd2]P[0x8c][0xf7][0xb8]>B[0xf1][0x84][0xd1]o[0xb2]DK[0xf7][0x98][0xbb]_V[0xb9][0xa8][0x1b][0x1][0xcc][0x4][0xb1]Z[0x0][0x87][0xfe][0xb5]1[0xc9]$![0x9f][0xe5][0x95][0xef][0xe7]#[0xe2],[0x1a][0xa9]Q2c`[0xc8][0xcb]FI[0xa8]U[0xaa]r[0xd2][0x99][0xd1][0xc7]<*[0x9f][0x9e][0x9e][0xcd][0xe6][0x1a][0x98][0xc3]e[0x10][0xc7]or<[0xef]5x|[0xb5][0xf5][0xa6][0xd4]j[\n]"
2022-02-09 12:33:15 [main] DEBUG org.apache.http.wire:87 - http-outgoing-0 >> "[0xd5][0xe8][0x17][0x11]{[0xfd][0xf9][0xc1]#S]_[0xfa][0xdb][0xfe][0xd1]6-[0xee]V4[0x81]W[0xeb][0x86][0xe0][0xeb]k[0xdb][0xef][0x84]c[0xb5]1[0xba][0xfc][0xb6]}[0x7][0x11]1[0xa0]"
2022-02-09 12:33:16 [main] DEBUG org.apache.http.wire:73 - http-outgoing-0 << "HTTP/1.1 400 Bad Request[\r][\n]"
2022-02-09 12:33:16 [main] DEBUG org.apache.http.wire:73 - http-outgoing-0 << "x-amz-request-id: AVWVWF79SFG0GBZZ[\r][\n]"
2022-02-09 12:33:16 [main] DEBUG org.apache.http.wire:73 - http-outgoing-0 << "x-amz-id-2: mJVXvAT1LJ3LzjtJ2oGI4Gqf1Y47xfkeUvqbHrpJSwOR8MiOTFzxOJ33RiAyYYnE41OdtV11OJg=[\r][\n]"
2022-02-09 12:33:16 [main] DEBUG org.apache.http.wire:73 - http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
2022-02-09 12:33:16 [main] DEBUG org.apache.http.wire:73 - http-outgoing-0 << "Transfer-Encoding: chunked[\r][\n]"
2022-02-09 12:33:16 [main] DEBUG org.apache.http.wire:73 - http-outgoing-0 << "Date: Wed, 09 Feb 2022 04:33:15 GMT[\r][\n]"
2022-02-09 12:33:16 [main] DEBUG org.apache.http.wire:73 - http-outgoing-0 << "Server: AmazonS3[\r][\n]"
2022-02-09 12:33:16 [main] DEBUG org.apache.http.wire:73 - http-outgoing-0 << "Connection: close[\r][\n]"
2022-02-09 12:33:16 [main] DEBUG org.apache.http.wire:73 - http-outgoing-0 << "[\r][\n]"
2022-02-09 12:33:16 [main] DEBUG org.apache.http.wire:73 - http-outgoing-0 << "110[\r][\n]"
2022-02-09 12:33:16 [main] DEBUG org.apache.http.wire:73 - http-outgoing-0 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
2022-02-09 12:33:16 [main] DEBUG org.apache.http.wire:73 - http-outgoing-0 << "<Error><Code>IncompleteBody</Code><Message>The request body terminated unexpectedly</Message><RequestId>AVWVWF79SFG0GBZZ</RequestId><HostId>mJVXvAT1LJ3LzjtJ2oGI4Gqf1Y47xfkeUvqbHrpJSwOR8MiOTFzxOJ33RiAyYYnE41OdtV11OJg=</HostId></Error>[\r][\n]"
2022-02-09 12:33:16 [main] DEBUG org.apache.http.wire:73 - http-outgoing-0 << "0[\r][\n]"
2022-02-09 12:33:16 [main] DEBUG org.apache.http.wire:73 - http-outgoing-0 << "[\r][\n]"
2022-02-09 12:33:16 [main] DEBUG software.amazon.awssdk.request:85 - Received failed response: 400, Request ID: AVWVWF79SFG0GBZZ, Extended Request ID: mJVXvAT1LJ3LzjtJ2oGI4Gqf1Y47xfkeUvqbHrpJSwOR8MiOTFzxOJ33RiAyYYnE41OdtV11OJg=

PutObject: Received an error response: software.amazon.awssdk.services.s3.model.S3Exception: The request body terminated unexpectedly (Service: S3, Status Code: 400, Request ID: AVWVWF79SFG0GBZZ, Extended Request ID: mJVXvAT1LJ3LzjtJ2oGI4Gqf1Y47xfkeUvqbHrpJSwOR8MiOTFzxOJ33RiAyYYnE41OdtV11OJg=)

software.amazon.awssdk.services.s3.model.S3Exception: The request body terminated unexpectedly (Service: S3, Status Code: 400, Request ID: AVWVWF79SFG0GBZZ, Extended Request ID: mJVXvAT1LJ3LzjtJ2oGI4Gqf1Y47xfkeUvqbHrpJSwOR8MiOTFzxOJ33RiAyYYnE41OdtV11OJg=)
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:108)
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:85)
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:43)
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler$Crc32ValidationResponseHandler.handle(AwsSyncClientHandler.java:95)
        at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$6(BaseClientHandler.java:234)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:81)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56)
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
        at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:167)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:82)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:175)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76)
        at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56)
        at software.amazon.awssdk.services.s3.DefaultS3Client.putObject(DefaultS3Client.java:8949)
        at chunk.tester.App.putObject(App.java:140)
        at chunk.tester.App.run(App.java:78)
        at picocli.CommandLine.executeUserObject(CommandLine.java:1939)
        at picocli.CommandLine.access$1300(CommandLine.java:145)
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
        at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
        at picocli.CommandLine.execute(CommandLine.java:2078)
        at chunk.tester.App.main(App.java:45)

tatsuya6502 · 2022-02-09T05:14:01Z

Please try to run the sample program with and without -H option:

This will fail:

$ ./gradlew run --args='--bucket mybucket -H'

...
PutObject: Received an error response: software.amazon.awssdk.services.s3.model.S3Exception: The request body terminated unexpectedly (Service: S3, Status Code: 400, Request ID: AVWVWF79SFG0GBZZ, Extended Request ID: mJVXvAT1LJ3LzjtJ2oGI4Gqf1Y47xfkeUvqbHrpJSwOR8MiOTFzxOJ33RiAyYYnE41OdtV11OJg=)
...
software.amazon.awssdk.services.s3.model.S3Exception: The request body terminated unexpectedly (Service: S3, Status Code: 400, Request ID: AVWVWF79SFG0GBZZ, Extended Request ID: mJVXvAT1LJ3LzjtJ2oGI4Gqf1Y47xfkeUvqbHrpJSwOR8MiOTFzxOJ33RiAyYYnE41OdtV11OJg=)
...

This will succeed:

$ ./gradlew run --args='--bucket mybucket'

...
PutObject: Sending a request. (key: key1, object size: 512, content length: 256)
...
2022-02-09 12:56:41 [main] DEBUG software.amazon.awssdk.request:85 - Received successful response: 200, Request ID: PC5CA6TBT1D3A2BT, Extended Request ID: H+aqKxQvQSyjIZZDPxR+tDnkUKxNinR7YIkKSrOe86Lg84QtUFOe5/fSDYGafQb7Itp0An0yL50=
...

The former (with -H) uses a custom endpoint wih HTTP protocol (port 80):

http://s3.ap-northeast-1.amazonaws.com (Tokyo region, ap-northeast-1)

The latter (without -H) uses the default endpoint with HTTPS protocol (port 443):

https://s3.ap-northeast-1.amazonaws.com (Tokyo region, ap-northeast-1)

By some reason, AWS SDK for Java 2 will not use chunked transfer encoding for PutObject when the default HTTPS protocol is used, and it will send a correct request.

The former (HTTP):

Will send a wrong request using chunked transfer encoding.

http-outgoing-0 >> "x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD[\r][\n]"
http-outgoing-0 >> "x-amz-decoded-content-length: 256[\r][\n]"
http-outgoing-0 >> "Content-Length: 430[\r][\n]"
http-outgoing-0 >> "200;chunk-signature=469b918c28d4a24ca832a5ead443ccce19ca05027cd3d6eaf0f01b0602373d46[\r][\n]"

0x200 (hexadecimal) = 512 (decimal)

The latter (HTTPS):

Will send a correct request by not using chunked transfer encoding.

http-outgoing-0 >> "x-amz-content-sha256: UNSIGNED-PAYLOAD[\r][\n]"
http-outgoing-0 >> "Content-Length: 256[\r][\n]"
http-outgoing-0 >> "7[0xf7][0x92]Y^2[0xc7][0xdb]Z[0xe1] ... "

aaoswal · 2022-02-22T21:14:31Z

Thank you @tatsuya6502 for your response.

The issue is reproducible using the steps mentioned above.

dagnir · 2022-03-14T23:40:46Z

This does look like a bug to me since it doesn't work with HTTP. Removing needs-review.

This commit ensures that when a RequestBody is created with an InputStream, the content-length also set on the RequestBody is honored, regardless of how much extra data is available in the stream. Note that if less data is actually available in the wrapped stream, this implmenetation will return EOF as well. Fixes aws#2908

This commit ensures that when a RequestBody is created with an InputStream, the content-length also set on the RequestBody is honored, regardless of how much extra data is available in the stream. Note that if less data is actually available in the wrapped stream, this implmenetation will return EOF as well. Fixes #2908

github-actions · 2022-03-29T19:57:00Z

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

…433c12aee Pull request: release <- staging/0efbc641-68ea-420b-94c4-c06433c12aee

tatsuya6502 added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Dec 10, 2021

aaoswal removed the needs-triage This issue or PR still needs to be triaged. label Feb 3, 2022

aaoswal self-assigned this Feb 3, 2022

aaoswal added the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 10 days. label Feb 3, 2022

github-actions bot added the closing-soon This issue will close in 4 days unless further comments are made. label Feb 8, 2022

github-actions bot removed closing-soon This issue will close in 4 days unless further comments are made. response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 10 days. labels Feb 9, 2022

aaoswal added the needs-review This issue or PR needs review from the team. label Feb 22, 2022

dagnir removed the needs-review This issue or PR needs review from the team. label Mar 14, 2022

debora-ito unassigned aaoswal Mar 23, 2022

dagnir mentioned this issue Mar 24, 2022

Honor content-length set on request body #3123

Merged

12 tasks

dagnir closed this as completed in #3123 Mar 29, 2022

aws-sdk-java-automation added a commit that referenced this issue Feb 26, 2024

Merge pull request #2908 from aws/staging/0efbc641-68ea-420b-94c4-c06…

48b4002

…433c12aee Pull request: release <- staging/0efbc641-68ea-420b-94c4-c06433c12aee

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

SDK ignores "Content-Length" header of a S3 PutObject request with the body backed by an InputStream #2908

SDK ignores "Content-Length" header of a S3 PutObject request with the body backed by an InputStream #2908

tatsuya6502 commented Dec 10, 2021 •

edited

Loading

aaoswal commented Feb 3, 2022 •

edited

Loading

Uh oh!

github-actions bot commented Feb 8, 2022

Uh oh!

tatsuya6502 commented Feb 9, 2022 •

edited

Loading

Uh oh!

tatsuya6502 commented Feb 9, 2022

Uh oh!

aaoswal commented Feb 22, 2022

Uh oh!

dagnir commented Mar 14, 2022

Uh oh!

github-actions bot commented Mar 29, 2022

Uh oh!

SDK ignores "Content-Length" header of a S3 PutObject request with the body backed by an InputStream #2908

SDK ignores "Content-Length" header of a S3 PutObject request with the body backed by an InputStream #2908

Comments

tatsuya6502 commented Dec 10, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe the bug

Expected behavior

Current behavior

Steps to Reproduce

Possible Solution

Context

AWS Java SDK version used

JDK version used

Operating System and version

aaoswal commented Feb 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 8, 2022

Uh oh!

tatsuya6502 commented Feb 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Full sample code for reproduction of the issue

Full stack trace of the error

Uh oh!

tatsuya6502 commented Feb 9, 2022

Uh oh!

aaoswal commented Feb 22, 2022

Uh oh!

dagnir commented Mar 14, 2022

Uh oh!

github-actions bot commented Mar 29, 2022

⚠️COMMENT VISIBILITY WARNING⚠️

Uh oh!

tatsuya6502 commented Dec 10, 2021 •

edited

Loading

aaoswal commented Feb 3, 2022 •

edited

Loading

tatsuya6502 commented Feb 9, 2022 •

edited

Loading