Skip to content

Commit 3d8e868

Browse files
committed
More details on testing
1 parent 0cdc55b commit 3d8e868

File tree

1 file changed

+21
-2
lines changed

1 file changed

+21
-2
lines changed

README.md

Lines changed: 21 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,28 @@
11
# aws-sdk-async-response-bytes
22

3+
_Reducing the memory consumption of AWS SDK for Java v2 when loading an S3 object into memory_
4+
5+
This repo is for evaluation of different approaches to reducing memory consumption of the AWS SDK Java v2 [async](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/asynchronous.html)
6+
method
7+
[`AsyncResponseTransformer.toBytes()`](https://github.com/aws/aws-sdk-java-v2/blob/69f7191252c26b351f7fb1c5f031948dac43e4c9/core/sdk-core/src/main/java/software/amazon/awssdk/core/async/AsyncResponseTransformer.java#L204-L206).
8+
The resulting PR is https://github.com/aws/aws-sdk-java-v2/pull/4355.
9+
10+
Approaches under consideration for reducing memory usage:
11+
12+
* Change A: Optimise byte storage, comes in two successive parts:
13+
* [**A1**](https://github.com/rtyley/aws-sdk-async-response-bytes/pull/8): Use the Content Length to initialise the `ByteArrayOutputStream` with **a byte array of the right size**.
14+
* [**A2**](https://github.com/rtyley/aws-sdk-async-response-bytes/pull/10): Use a simple fixed-size byte array in preference to a `ByteArrayOutputStream`
15+
* Change [**B**](https://github.com/rtyley/aws-sdk-async-response-bytes/pull/12): Avoid performing a byte array copy when creating the `ResponseBytes` instance
16+
17+
Approaches A & B can be considered individually, but work much better when combined.
18+
19+
## Automated Memory-Consumption tests
20+
321
The test script [`findMinMem.sh`](https://github.com/rtyley/aws-sdk-async-response-bytes/blob/ab9307fee6d4965cce0cba60c439035a22cf2d98/findMinMem.sh)
422
repeatedly downloads a [**258 MB**](https://github.com/rtyley/aws-sdk-async-response-bytes/blob/eb84c15259f947e91c442eb0b8eda68b8cdbefbb/src/main/java/com/madgag/aws/sdk/async/responsebytes/KnownS3Object.java#L7)
5-
object from S3 into memory, while varying the amount of Java heap memory allocated with `-Xmx`,
6-
to find the amount of memory necessary for the download to consistently succeed.
23+
object from S3 into memory, while varying the amount of Java heap memory allocated with `-Xmx`,
24+
to find the amount of memory necessary for the download to consistently succeed with the given
25+
approach.
726

827
These are the resulting memory requirements found, in MB, for all possible permutations
928
of the fixes:

0 commit comments

Comments
 (0)