|
1 | 1 | # aws-sdk-async-response-bytes
|
2 | 2 |
|
| 3 | +_Reducing the memory consumption of AWS SDK for Java v2 when loading an S3 object into memory_ |
| 4 | + |
| 5 | +This repo is for evaluation of different approaches to reducing memory consumption of the AWS SDK Java v2 [async](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/asynchronous.html) |
| 6 | +method |
| 7 | +[`AsyncResponseTransformer.toBytes()`](https://github.com/aws/aws-sdk-java-v2/blob/69f7191252c26b351f7fb1c5f031948dac43e4c9/core/sdk-core/src/main/java/software/amazon/awssdk/core/async/AsyncResponseTransformer.java#L204-L206). |
| 8 | +The resulting PR is https://github.com/aws/aws-sdk-java-v2/pull/4355. |
| 9 | + |
| 10 | +Approaches under consideration for reducing memory usage: |
| 11 | + |
| 12 | +* Change A: Optimise byte storage, comes in two successive parts: |
| 13 | + * [**A1**](https://github.com/rtyley/aws-sdk-async-response-bytes/pull/8): Use the Content Length to initialise the `ByteArrayOutputStream` with **a byte array of the right size**. |
| 14 | + * [**A2**](https://github.com/rtyley/aws-sdk-async-response-bytes/pull/10): Use a simple fixed-size byte array in preference to a `ByteArrayOutputStream` |
| 15 | +* Change [**B**](https://github.com/rtyley/aws-sdk-async-response-bytes/pull/12): Avoid performing a byte array copy when creating the `ResponseBytes` instance |
| 16 | + |
| 17 | +Approaches A & B can be considered individually, but work much better when combined. |
| 18 | + |
| 19 | +## Automated Memory-Consumption tests |
| 20 | + |
3 | 21 | The test script [`findMinMem.sh`](https://github.com/rtyley/aws-sdk-async-response-bytes/blob/ab9307fee6d4965cce0cba60c439035a22cf2d98/findMinMem.sh)
|
4 | 22 | repeatedly downloads a [**258 MB**](https://github.com/rtyley/aws-sdk-async-response-bytes/blob/eb84c15259f947e91c442eb0b8eda68b8cdbefbb/src/main/java/com/madgag/aws/sdk/async/responsebytes/KnownS3Object.java#L7)
|
5 |
| -object from S3 into memory, while varying the amount of Java heap memory allocated with `-Xmx`, |
6 |
| -to find the amount of memory necessary for the download to consistently succeed. |
| 23 | +object from S3 into memory, while varying the amount of Java heap memory allocated with `-Xmx`, |
| 24 | +to find the amount of memory necessary for the download to consistently succeed with the given |
| 25 | +approach. |
7 | 26 |
|
8 | 27 | These are the resulting memory requirements found, in MB, for all possible permutations
|
9 | 28 | of the fixes:
|
|
0 commit comments