Skip to content

Commit 7d82072

Browse files
authored
fix(s3-deployment): optimize memory usage for large files (#34020)
### Issue # (if applicable) Closes #34002. ### Reason for this change The original fix for issue #22661 (PR #33698) introduced a regression where the S3 deployment Lambda would read entire files into memory to check if they're JSON. This approach works fine for small files but causes Lambda timeouts and memory issues with large files (10MB+). This is particularly problematic for customers deploying large assets to S3 buckets. ### Description of changes The S3 deployment Lambda handler was reading entire files into memory to check if they're JSON, causing timeouts and memory issues with large files (10MB+). This change optimizes the S3 deployment Lambda handler to process files more efficiently by: 1. Adding an early return when there are no markers to replace 2. Processing all files line by line, which is much more memory-efficient than loading the full JSON in memory 3. Adding an optional `escape` parameter to the `Source.jsonData` method in order to control JSON escaping 4. Using the `jsonEscape` flag in `MarkersConfig` to control when special JSON escaping is needed These changes ensure that: - Files without markers are processed instantly - Files with markers are processed line by line, minimizing memory usage - Special JSON escaping is only applied when explicitly requested The implementation is backward compatible with the experience before the PR #33698 was merged, as it maintains the existing behavior by default but provides an opt-in mechanism for JSON escaping when needed. The opt-in mechanism is required for users who were benefitting from the escaping mechanism introduced by the PR #33698. ### Describe any new or updated permissions being added No new or updated IAM permissions are required for this change. ### Description of how you validated changes - Created an integration test (`integ.bucket-deployment-large-file.ts`) that reproduces the issue with large files - Implemented local testing to verify the fix with both small and large files - Added memory limit assertions to ensure memory usage stays within acceptable bounds - Conducted performance testing with isolated test runs to measure memory usage across various file types and sizes The integration test specifically validates that large files (10MB+) can be successfully deployed without memory issues, ensuring the fix works in real-world scenarios. ### Checklist - [ ] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md) ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
1 parent ccd8de7 commit 7d82072

File tree

600 files changed

+487184
-84489
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

600 files changed

+487184
-84489
lines changed

.gitattributes

+1
Original file line numberDiff line numberDiff line change
@@ -17,3 +17,4 @@ packages/@aws-cdk-testing/framework-integ/test/aws-s3-deployment/test/integ.buck
1717
packages/@aws-cdk-testing/framework-integ/test/**/*.snapshot/**/asset*.zip filter=lfs diff=lfs merge=lfs -text
1818
packages/@aws-cdk/*-alpha/test/**/*.snapshot/**/asset*.zip filter=lfs diff=lfs merge=lfs -text
1919
packages/@aws-cdk/*-alpha/test/*.snapshot/asset.*/bootstrap filter=lfs diff=lfs merge=lfs -text
20+
packages/@aws-cdk-testing/framework-integ/test/aws-s3-deployment/test/integ.bucket-deployment-large-file/asset*/large* filter=lfs diff=lfs merge=lfs -text

packages/@aws-cdk-testing/framework-integ/test/aws-appconfig/test/integ.configuration.js.snapshot/appconfigconfigurationDefaultTestDeployAssert6752CD38.assets.json

+2-1
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

packages/@aws-cdk-testing/framework-integ/test/aws-appconfig/test/integ.configuration.js.snapshot/asset.c6358465bf49dfae556bb430bf9c81fa578c221b82c308e3707901b1dd654762/index.py

-336
This file was deleted.

packages/@aws-cdk-testing/framework-integ/test/aws-appconfig/test/integ.configuration.js.snapshot/asset.e42a736be21cd3134b9bff4e71e3afa99a4cc900ae489e9a7f7025c8d258f9b8.zip

-3
This file was deleted.

0 commit comments

Comments
 (0)