Skip to content

Pre-compile all python code and remove original .py files. #476

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Apr 22, 2024
Merged

Pre-compile all python code and remove original .py files. #476

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
b78777d
Remove urllib3 dist-info files as well.
purple4reina Apr 17, 2024
6470ea6
Pre-compile all python code and remove original .py files.
purple4reina Apr 17, 2024
1df454f
Compress at optimization level 2.
purple4reina Apr 17, 2024
b9eb735
Add PYTHONNODEBUGRANGES=1 to compile step.
purple4reina Apr 18, 2024
34a3dc0
Add comment.
purple4reina Apr 19, 2024
22e8068
Use different commandline option to support py3.8
purple4reina Apr 19, 2024
4933d7e
Update size limits.
purple4reina Apr 19, 2024
f0afead
Leave __init__.py contrib files.
purple4reina Apr 22, 2024
582e612
Remove text like files.
purple4reina Apr 22, 2024
43947e9
oops
purple4reina Apr 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 13 additions & 4 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,23 +11,32 @@ WORKDIR /build
COPY . .
RUN pip install . -t ./python/lib/$runtime/site-packages

# Remove *.pyc files
RUN find ./python/lib/$runtime/site-packages -name \*.pyc -delete

# Remove botocore (40MB) to reduce package size. aws-xray-sdk
# installs it, while it's already provided by the Lambda Runtime.
RUN rm -rf ./python/lib/$runtime/site-packages/botocore*
RUN rm -rf ./python/lib/$runtime/site-packages/setuptools
RUN rm -rf ./python/lib/$runtime/site-packages/jsonschema/tests
RUN find . -name 'libddwaf.so' -delete
RUN rm -rf ./python/lib/$runtime/site-packages/urllib3
RUN rm -rf ./python/lib/$runtime/site-packages/urllib3*
RUN rm ./python/lib/$runtime/site-packages/ddtrace/appsec/_iast/_taint_tracking/*.so
RUN rm ./python/lib/$runtime/site-packages/ddtrace/appsec/_iast/_stacktrace*.so
RUN rm ./python/lib/$runtime/site-packages/ddtrace/internal/datadog/profiling/libdd_wrapper.so
RUN rm ./python/lib/$runtime/site-packages/ddtrace/internal/datadog/profiling/ddup/_ddup.*.so
RUN rm ./python/lib/$runtime/site-packages/ddtrace/internal/datadog/profiling/stack_v2/_stack_v2.*.so
RUN find . -name "*.dist-info" -type d | xargs rm -rf

# Precompile all .pyc files and remove .py files. This speeds up load time.
# Compile with optimization level 2 (-OO) and PYTHONNODEBUGRANGES=1 to redtce
# size of .pyc files.
# See https://docs.python.org/3/tutorial/modules.html#compiled-python-files
# https://docs.python.org/3.11/using/cmdline.html#cmdoption-O
# https://docs.python.org/3/using/cmdline.html#envvar-PYTHONNODEBUGRANGES
RUN PYTHONNODEBUGRANGES=1 python -OO -m compileall -b ./python/lib/$runtime/site-packages
# remove all .py files except ddtrace/contrib/*/__init__.py which are necessary
# for ddtrace.patch to discover instrumationation packages.
RUN find ./python/lib/$runtime/site-packages -name \*.py | grep -v ddtrace/contrib | xargs rm -rf
RUN find ./python/lib/$runtime/site-packages/ddtrace/contrib -name \*.py | grep -v __init__ | xargs rm -rf
RUN find ./python/lib/$runtime/site-packages -name __pycache__ -type d -exec rm -r {} \+

FROM scratch
COPY --from=builder /build/python /
5 changes: 2 additions & 3 deletions scripts/check_layer_size.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,9 @@

# Compares layer size to threshold, and fails if below that threshold

# 7 mb size limit
set -e
MAX_LAYER_COMPRESSED_SIZE_KB=$(expr 4 \* 1024)
MAX_LAYER_UNCOMPRESSED_SIZE_KB=$(expr 13 \* 1024)
MAX_LAYER_COMPRESSED_SIZE_KB=$(expr 5 \* 1024)
MAX_LAYER_UNCOMPRESSED_SIZE_KB=$(expr 12 \* 1024)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bytecode is harder to compress (hence increase from 4->5) but uncompressed is much smaller (hence decrease from 13->12)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good! Could we update this changes in serverless-tools once this is merged? Thanks!



LAYER_FILES_PREFIX="datadog_lambda_py"
Expand Down
Loading