feat(lambda-python): support for providing a custom bundling docker image (#18082)

This refactors the bundling process to match the NodeJs and Go Lambda functions and allows providing a custom bundling docker image.

Changes:
- refactor bundling to use `cdk.BundlingOptions`
- Use updated `Bundling` class
- Update tests to use updated `Bundling` class


Fixes #10298, #12949, #15391, #16234, #15306

BREAKING CHANGE: `assetHashType` and `assetHash` properties moved to new `bundling` property.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

Author: setu4993

packages/@aws-cdk/aws-lambda-python/README.md

@@ -27,29 +27,61 @@ Define a `PythonFunction`:
 ```ts
 new lambda.PythonFunction(this, 'MyFunction', {
   entry: '/path/to/my/function', // required
-  runtime: Runtime.PYTHON_3_6, // required
+  runtime: Runtime.PYTHON_3_8, // required
   index: 'my_index.py', // optional, defaults to 'index.py'
   handler: 'my_exported_func', // optional, defaults to 'handler'
 });
 ```
 
 All other properties of `lambda.Function` are supported, see also the [AWS Lambda construct library](https://github.com/aws/aws-cdk/tree/master/packages/%40aws-cdk/aws-lambda).
 
-## Module Dependencies
+## Python Layer
 
-If `requirements.txt` or `Pipfile` exists at the entry path, the construct will handle installing
-all required modules in a [Lambda compatible Docker container](https://gallery.ecr.aws/sam/build-python3.7)
-according to the `runtime` and with the Docker platform based on the target architecture of the Lambda function.
+You may create a python-based lambda layer with `PythonLayerVersion`. If `PythonLayerVersion` detects a `requirements.txt`
+or `Pipfile` or `poetry.lock` with the associated `pyproject.toml` at the entry path, then `PythonLayerVersion` will include the dependencies inline with your code in the
+layer.
+
+Define a `PythonLayerVersion`:
+
+```ts
+new lambda.PythonLayerVersion(this, 'MyLayer', {
+  entry: '/path/to/my/layer', // point this to your library's directory
+})
+```
+
+A layer can also be used as a part of a `PythonFunction`:
+
+```ts
+new lambda.PythonFunction(this, 'MyFunction', {
+  entry: '/path/to/my/function',
+  runtime: Runtime.PYTHON_3_8,
+  layers: [
+    new lambda.PythonLayerVersion(this, 'MyLayer', {
+      entry: '/path/to/my/layer', // point this to your library's directory
+    }),
+  ],
+});
+```
+
+## Packaging
+
+If `requirements.txt`, `Pipfile` or `poetry.lock` exists at the entry path, the construct will handle installing all required modules in a [Lambda compatible Docker container](https://gallery.ecr.aws/sam/build-python3.7) according to the `runtime` and with the Docker platform based on the target architecture of the Lambda function.
 
 Python bundles are only recreated and published when a file in a source directory has changed.
 Therefore (and as a general best-practice), it is highly recommended to commit a lockfile with a
-list of all transitive dependencies and their exact versions.
-This will ensure that when any dependency version is updated, the bundle asset is recreated and uploaded.
+list of all transitive dependencies and their exact versions. This will ensure that when any dependency version is updated, the bundle asset is recreated and uploaded.
+
+To that end, we recommend using [`pipenv`] or [`poetry`] which have lockfile support.
+
+- [`pipenv`](https://pipenv-fork.readthedocs.io/en/latest/basics.html#example-pipfile-lock)
+- [`poetry`](https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control)
 
-To that end, we recommend using [`pipenv`] or [`poetry`] which has lockfile support.
+Packaging is executed using the `Packaging` class, which:
 
-[`pipenv`]: https://pipenv-fork.readthedocs.io/en/latest/basics.html#example-pipfile-lock
-[`poetry`]: https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+1. Infers the packaging type based on the files present.
+2. If it sees a `Pipfile` or a `poetry.lock` file, it exports it to a compatible `requirements.txt` file with credentials (if they're available in the source files or in the bundling container).
+3. Installs dependencies using `pip`.
+4. Copies the dependencies into an asset that is bundled for the Lambda package.
 
 **Lambda with a requirements.txt**
 
@@ -73,24 +105,71 @@ To that end, we recommend using [`pipenv`] or [`poetry`] which has lockfile supp
 ```plaintext
 .
 ├── lambda_function.py # exports a function named 'handler'
-├── pyproject.toml # has to be present at the entry path
-├── poetry.lock # your poetry lock file
+├── pyproject.toml # your poetry project definition
+├── poetry.lock # your poetry lock file has to be present at the entry path
 ```
 
-**Lambda Layer Support**
+## Custom Bundling
 
-You may create a python-based lambda layer with `PythonLayerVersion`. If `PythonLayerVersion` detects a `requirements.txt`
-or `Pipfile` or `poetry.lock` with the associated `pyproject.toml` at the entry path, then `PythonLayerVersion` will include the dependencies inline with your code in the
-layer.
+Custom bundling can be performed by passing in additional build arguments that point to index URLs to private repos, or by using an entirely custom Docker images for bundling dependencies. The build args currently supported are:
+
+- `PIP_INDEX_URL`
+- `PIP_EXTRA_INDEX_URL`
+- `HTTPS_PROXY`
+
+Additional build args for bundling that refer to PyPI indexes can be specified as:
 
 ```ts
-new lambda.PythonFunction(this, 'MyFunction', {
-  entry: '/path/to/my/function',
-  runtime: Runtime.PYTHON_3_6,
-  layers: [
-    new lambda.PythonLayerVersion(this, 'MyLayer', {
-      entry: '/path/to/my/layer', // point this to your library's directory
-    }),
-  ],
+const entry = '/path/to/function';
+const image = DockerImage.fromBuild(entry);
+
+new lambda.PythonFunction(this, 'function', {
+  entry,
+  runtime: Runtime.PYTHON_3_8,
+  bundling: {
+    buildArgs: { PIP_INDEX_URL: "https://your.index.url/simple/", PIP_EXTRA_INDEX_URL: "https://your.extra-index.url/simple/" },
+  },
 });
 ```
+
+If using a custom Docker image for bundling, the dependencies are installed with `pip`, `pipenv` or `poetry` by using the `Packaging` class. A different bundling Docker image that is in the same directory as the function can be specified as:
+
+ ```ts
+const entry = '/path/to/function';
+const image = DockerImage.fromBuild(entry);
+
+new lambda.PythonFunction(this, 'function', {
+  entry,
+  runtime: Runtime.PYTHON_3_8,
+  bundling: { image },
+});
+```
+
+## Custom Bundling with Code Artifact
+
+To use a Code Artifact PyPI repo, the `PIP_INDEX_URL` for bundling the function can be customized (requires AWS CLI in the build environment):
+
+```ts
+import { execSync } from 'child_process';
+
+const entry = '/path/to/function';
+const image = DockerImage.fromBuild(entry);
+
+const domain = 'my-domain';
+const domainOwner = '111122223333';
+const repoName = 'my_repo';
+const region = 'us-east-1';
+const codeArtifactAuthToken = execSync(`aws codeartifact get-authorization-token --domain ${domain} --domain-owner ${domainOwner} --query authorizationToken --output text`).toString().trim();
+
+const indexUrl = `https://aws:${codeArtifactAuthToken}@${domain}-${domainOwner}.d.codeartifact.${region}.amazonaws.com/pypi/${repoName}/simple/`;
+
+new lambda.PythonFunction(this, 'function', {
+  entry,
+  runtime: Runtime.PYTHON_3_8,
+  bundling: {
+    buildArgs: { PIP_INDEX_URL: indexUrl },
+  },
+});
+```
+
+This type of an example should work for `pip` and `poetry` based dependencies, but will not work for `pipenv`.

packages/@aws-cdk/aws-lambda-python/lib/Dockerfile

@@ -3,7 +3,12 @@
 ARG IMAGE=public.ecr.aws/sam/build-python3.7
 FROM $IMAGE
 
-# Ensure rsync is installed
-RUN yum -q list installed rsync &>/dev/null || yum install -y rsync
+ARG PIP_INDEX_URL
+ARG PIP_EXTRA_INDEX_URL
+ARG HTTPS_PROXY
+
+# Upgrade pip (required by cryptography v3.4 and above, which is a dependency of poetry)
+RUN pip install --upgrade pip
+RUN pip install pipenv poetry
 
 CMD [ "python" ]