Merge branch 'develop' into chore/pre-release

Author: heitorlessa

.github/workflows/quality_check.yml

@@ -68,6 +68,8 @@ jobs:
         run: make mypy
       - name: Test with pytest
         run: make test
+      - name: Test dependencies with Nox
+        run: make test-dependencies
       - name: Security baseline
         run: make security-baseline
       - name: Complexity baseline

.github/workflows/release.yml

@@ -237,12 +237,12 @@ jobs:
 
       - name: Upload to PyPi prod
         if: ${{ !inputs.skip_pypi }}
-        uses: pypa/gh-action-pypi-publish@81e9d935c883d0b210363ab89cf05f3894778450 # v1.8.14
+        uses: pypa/gh-action-pypi-publish@ec4db0b4ddc65acdf4bff5fa45ac92d78b56bdf0 # v1.9.0
 
       # PyPi test maintenance affected us numerous times, leaving for history purposes
       # - name: Upload to PyPi test
       #   if: ${{ !inputs.skip_pypi }}
-      #   uses: pypa/gh-action-pypi-publish@81e9d935c883d0b210363ab89cf05f3894778450 # v1.8.14
+      #   uses: pypa/gh-action-pypi-publish@ec4db0b4ddc65acdf4bff5fa45ac92d78b56bdf0 # v1.9.0
       #   with:
       #     repository-url: https://test.pypi.org/legacy/
 

.github/workflows/secure_workflows.yml

@@ -32,7 +32,7 @@ jobs:
       - name: Checkout code
         uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332  # v4.1.7
       - name: Ensure 3rd party workflows have SHA pinned
-        uses: zgosalvez/github-actions-ensure-sha-pinned-actions@76d1d8e0b075d7190b5d59b86da91c7bdbcc99b2 # v3.0.7
+        uses: zgosalvez/github-actions-ensure-sha-pinned-actions@74606c30450304eee8660aae751818321754feb1 # v3.0.9
         with:
           allowlist: |
             slsa-framework/slsa-github-generator

CHANGELOG.md

@@ -8,19 +8,54 @@
 
 * **event_sources:** change partition and offset field types in KafkaEventRecord ([#4515](https://github.com/aws-powertools/powertools-lambda-python/issues/4515))
 
+## Documentation
+
+* **i-made-this:** add new article about best practices for accelerating serverless development ([#4518](https://github.com/aws-powertools/powertools-lambda-python/issues/4518))
+
+## Features
+
+* **validation:** support JSON Schema referencing in validation utils ([#4508](https://github.com/aws-powertools/powertools-lambda-python/issues/4508))
+
 ## Maintenance
 
-* **deps:** bump redis from 5.0.5 to 5.0.6 ([#4527](https://github.com/aws-powertools/powertools-lambda-python/issues/4527))
+* **ci:** introduce tests with Nox ([#4537](https://github.com/aws-powertools/powertools-lambda-python/issues/4537))
+* **ci:** add the Data Masking feature to nox tests ([#4574](https://github.com/aws-powertools/powertools-lambda-python/issues/4574))
+* **ci:** add the Typing feature to nox tests ([#4572](https://github.com/aws-powertools/powertools-lambda-python/issues/4572))
+* **ci:** add the Validation feature to nox tests ([#4571](https://github.com/aws-powertools/powertools-lambda-python/issues/4571))
+* **ci:** add the Feature Flags feature to nox tests ([#4570](https://github.com/aws-powertools/powertools-lambda-python/issues/4570))
+* **ci:** add the Parameters feature to nox tests ([#4569](https://github.com/aws-powertools/powertools-lambda-python/issues/4569))
+* **ci:** add the Middleware Factory feature to nox tests ([#4568](https://github.com/aws-powertools/powertools-lambda-python/issues/4568))
+* **ci:** add the Metrics feature to nox tests ([#4552](https://github.com/aws-powertools/powertools-lambda-python/issues/4552))
+* **ci:** add the Tracer feature to nox tests ([#4567](https://github.com/aws-powertools/powertools-lambda-python/issues/4567))
+* **deps:** bump the layer-balancer group in /layer/scripts/layer-balancer with 3 updates ([#4565](https://github.com/aws-powertools/powertools-lambda-python/issues/4565))
 * **deps:** bump codecov/codecov-action from 4.4.1 to 4.5.0 ([#4514](https://github.com/aws-powertools/powertools-lambda-python/issues/4514))
-* **deps-dev:** bump cdklabs-generative-ai-cdk-constructs from 0.1.182 to 0.1.184 ([#4529](https://github.com/aws-powertools/powertools-lambda-python/issues/4529))
+* **deps:** bump squidfunk/mkdocs-material from `96abcbb` to `257eca8` in /docs ([#4540](https://github.com/aws-powertools/powertools-lambda-python/issues/4540))
+* **deps:** bump the layer-balancer group in /layer/scripts/layer-balancer with 3 updates ([#4546](https://github.com/aws-powertools/powertools-lambda-python/issues/4546))
+* **deps:** bump pypa/gh-action-pypi-publish from 1.8.14 to 1.9.0 ([#4538](https://github.com/aws-powertools/powertools-lambda-python/issues/4538))
+* **deps:** bump zgosalvez/github-actions-ensure-sha-pinned-actions from 3.0.7 to 3.0.9 ([#4539](https://github.com/aws-powertools/powertools-lambda-python/issues/4539))
+* **deps:** bump fastjsonschema from 2.19.1 to 2.20.0 ([#4543](https://github.com/aws-powertools/powertools-lambda-python/issues/4543))
+* **deps:** bump redis from 5.0.5 to 5.0.6 ([#4527](https://github.com/aws-powertools/powertools-lambda-python/issues/4527))
+* **deps-dev:** bump urllib3 from 1.26.18 to 1.26.19 in /layer ([#4547](https://github.com/aws-powertools/powertools-lambda-python/issues/4547))
+* **deps-dev:** bump aws-cdk-aws-lambda-python-alpha from 2.145.0a0 to 2.146.0a0 ([#4542](https://github.com/aws-powertools/powertools-lambda-python/issues/4542))
+* **deps-dev:** bump mypy-boto3-secretsmanager from 1.34.125 to 1.34.128 in the boto-typing group ([#4541](https://github.com/aws-powertools/powertools-lambda-python/issues/4541))
+* **deps-dev:** bump mkdocs-material from 9.5.26 to 9.5.27 ([#4544](https://github.com/aws-powertools/powertools-lambda-python/issues/4544))
+* **deps-dev:** bump cfn-lint from 0.87.7 to 1.3.0 ([#4577](https://github.com/aws-powertools/powertools-lambda-python/issues/4577))
+* **deps-dev:** bump cdklabs-generative-ai-cdk-constructs from 0.1.188 to 0.1.189 ([#4564](https://github.com/aws-powertools/powertools-lambda-python/issues/4564))
+* **deps-dev:** bump hvac from 2.2.0 to 2.3.0 ([#4563](https://github.com/aws-powertools/powertools-lambda-python/issues/4563))
+* **deps-dev:** bump filelock from 3.15.1 to 3.15.3 ([#4576](https://github.com/aws-powertools/powertools-lambda-python/issues/4576))
+* **deps-dev:** bump ruff from 0.4.8 to 0.4.9 ([#4528](https://github.com/aws-powertools/powertools-lambda-python/issues/4528))
 * **deps-dev:** bump aws-cdk-lib from 2.145.0 to 2.146.0 ([#4526](https://github.com/aws-powertools/powertools-lambda-python/issues/4526))
-* **deps-dev:** bump cfn-lint from 0.87.6 to 0.87.7 ([#4513](https://github.com/aws-powertools/powertools-lambda-python/issues/4513))
 * **deps-dev:** bump filelock from 3.14.0 to 3.15.1 ([#4512](https://github.com/aws-powertools/powertools-lambda-python/issues/4512))
-* **deps-dev:** bump cdklabs-generative-ai-cdk-constructs from 0.1.179 to 0.1.182 ([#4510](https://github.com/aws-powertools/powertools-lambda-python/issues/4510))
 * **deps-dev:** bump aws-cdk from 2.145.0 to 2.146.0 ([#4525](https://github.com/aws-powertools/powertools-lambda-python/issues/4525))
-* **deps-dev:** bump mypy-boto3-secretsmanager from 1.34.109 to 1.34.125 in the boto-typing group ([#4509](https://github.com/aws-powertools/powertools-lambda-python/issues/4509))
-* **deps-dev:** bump ruff from 0.4.8 to 0.4.9 ([#4528](https://github.com/aws-powertools/powertools-lambda-python/issues/4528))
+* **deps-dev:** bump cdklabs-generative-ai-cdk-constructs from 0.1.182 to 0.1.184 ([#4529](https://github.com/aws-powertools/powertools-lambda-python/issues/4529))
 * **deps-dev:** bump bandit from 1.7.8 to 1.7.9 ([#4511](https://github.com/aws-powertools/powertools-lambda-python/issues/4511))
+* **deps-dev:** bump cfn-lint from 0.87.6 to 0.87.7 ([#4513](https://github.com/aws-powertools/powertools-lambda-python/issues/4513))
+* **deps-dev:** bump cdklabs-generative-ai-cdk-constructs from 0.1.189 to 0.1.192 ([#4578](https://github.com/aws-powertools/powertools-lambda-python/issues/4578))
+* **deps-dev:** bump cdklabs-generative-ai-cdk-constructs from 0.1.179 to 0.1.182 ([#4510](https://github.com/aws-powertools/powertools-lambda-python/issues/4510))
+* **deps-dev:** bump sentry-sdk from 2.5.1 to 2.6.0 ([#4579](https://github.com/aws-powertools/powertools-lambda-python/issues/4579))
+* **deps-dev:** bump mypy-boto3-secretsmanager from 1.34.109 to 1.34.125 in the boto-typing group ([#4509](https://github.com/aws-powertools/powertools-lambda-python/issues/4509))
+* **deps-dev:** bump cdklabs-generative-ai-cdk-constructs from 0.1.184 to 0.1.188 ([#4550](https://github.com/aws-powertools/powertools-lambda-python/issues/4550))
+* **governance:** fix errors when creating Gitpod environment ([#4532](https://github.com/aws-powertools/powertools-lambda-python/issues/4532))
 
 
 <a name="v2.39.1"></a>

CONTRIBUTING.md

@@ -67,7 +67,7 @@ timeline
     Pre-Pull Request <br> (make pr)     : Code linting
                                         : Docs linting
                                         : Static typing analysis
-                                        : Tests (unit|functional|perf)
+                                        : Tests (unit|functional|perf|dependencies)
                                         : Security baseline
                                         : Complexity baseline
                                         : +pre-commit checks

Makefile

@@ -32,6 +32,9 @@ test:
 	poetry run pytest -m "not perf" --ignore tests/e2e --cov=aws_lambda_powertools --cov-report=xml
 	poetry run pytest --cache-clear tests/performance
 
+test-dependencies:
+	poetry run nox --error-on-external-run --reuse-venv=yes --non-interactive
+
 test-pydanticv2:
 	poetry run pytest -m "not perf" --ignore tests/e2e
 
@@ -83,7 +86,7 @@ complexity-baseline:
 	$(info Maintenability index)
 	poetry run radon mi aws_lambda_powertools
 	$(info Cyclomatic complexity index)
-	poetry run xenon --max-absolute C --max-modules A --max-average A aws_lambda_powertools --exclude aws_lambda_powertools/shared/json_encoder.py
+	poetry run xenon --max-absolute C --max-modules A --max-average A aws_lambda_powertools --exclude aws_lambda_powertools/shared/json_encoder.py,aws_lambda_powertools/utilities/validation/base.py
 
 #
 # Use `poetry version <major>/<minor></patch>` for version bump

aws_lambda_powertools/utilities/validation/base.py

@@ -3,12 +3,18 @@
 
 import fastjsonschema  # type: ignore
 
-from .exceptions import InvalidSchemaFormatError, SchemaValidationError
+from aws_lambda_powertools.utilities.validation.exceptions import InvalidSchemaFormatError, SchemaValidationError
 
 logger = logging.getLogger(__name__)
 
 
-def validate_data_against_schema(data: Union[Dict, str], schema: Dict, formats: Optional[Dict] = None):
+def validate_data_against_schema(
+    data: Union[Dict, str],
+    schema: Dict,
+    formats: Optional[Dict] = None,
+    handlers: Optional[Dict] = None,
+    provider_options: Optional[Dict] = None,
+):
     """Validate dict data against given JSON Schema
 
     Parameters
@@ -19,6 +25,11 @@ def validate_data_against_schema(data: Union[Dict, str], schema: Dict, formats:
         JSON Schema to validate against
     formats: Dict
         Custom formats containing a key (e.g. int64) and a value expressed as regex or callback returning bool
+    handlers: Dict
+        Custom methods to retrieve remote schemes, keyed off of URI scheme
+    provider_options: Dict
+        Arguments that will be passed directly to the underlying validation call, in this case fastjsonchema.validate.
+        For all supported arguments see: https://horejsek.github.io/python-fastjsonschema/#fastjsonschema.validate
 
     Raises
     ------
@@ -29,7 +40,9 @@ def validate_data_against_schema(data: Union[Dict, str], schema: Dict, formats:
     """
     try:
         formats = formats or {}
-        fastjsonschema.validate(definition=schema, data=data, formats=formats)
+        handlers = handlers or {}
+        provider_options = provider_options or {}
+        fastjsonschema.validate(definition=schema, data=data, formats=formats, handlers=handlers, **provider_options)
     except (TypeError, AttributeError, fastjsonschema.JsonSchemaDefinitionException) as e:
         raise InvalidSchemaFormatError(f"Schema received: {schema}, Formats: {formats}. Error: {e}")
     except fastjsonschema.JsonSchemaValueException as e:

aws_lambda_powertools/utilities/validation/exceptions.py

@@ -1,6 +1,6 @@
 from typing import Any, List, Optional
 
-from ...exceptions import InvalidEnvelopeExpressionError
+from aws_lambda_powertools.exceptions import InvalidEnvelopeExpressionError
 
 
 class SchemaValidationError(Exception):

aws_lambda_powertools/utilities/validation/validator.py

@@ -1,10 +1,9 @@
 import logging
 from typing import Any, Callable, Dict, Optional, Union
 
+from aws_lambda_powertools.middleware_factory import lambda_handler_decorator
 from aws_lambda_powertools.utilities import jmespath_utils
-
-from ...middleware_factory import lambda_handler_decorator
-from .base import validate_data_against_schema
+from aws_lambda_powertools.utilities.validation.base import validate_data_against_schema
 
 logger = logging.getLogger(__name__)
 
@@ -16,8 +15,12 @@ def validator(
     context: Any,
     inbound_schema: Optional[Dict] = None,
     inbound_formats: Optional[Dict] = None,
+    inbound_handlers: Optional[Dict] = None,
+    inbound_provider_options: Optional[Dict] = None,
     outbound_schema: Optional[Dict] = None,
     outbound_formats: Optional[Dict] = None,
+    outbound_handlers: Optional[Dict] = None,
+    outbound_provider_options: Optional[Dict] = None,
     envelope: str = "",
     jmespath_options: Optional[Dict] = None,
     **kwargs: Any,
@@ -44,6 +47,17 @@ def validator(
         Custom formats containing a key (e.g. int64) and a value expressed as regex or callback returning bool
     outbound_formats: Dict
         Custom formats containing a key (e.g. int64) and a value expressed as regex or callback returning bool
+    inbound_handlers: Dict
+        Custom methods to retrieve remote schemes, keyed off of URI scheme
+    outbound_handlers: Dict
+        Custom methods to retrieve remote schemes, keyed off of URI scheme
+    inbound_provider_options: Dict
+        Arguments that will be passed directly to the underlying validation call, in this case fastjsonchema.validate.
+        For all supported arguments see: https://horejsek.github.io/python-fastjsonschema/#fastjsonschema.validate
+    outbound_provider_options: Dict
+        Arguments that will be passed directly to the underlying validation call, in this case fastjsonchema.validate.
+        For all supported arguments see: https://horejsek.github.io/python-fastjsonschema/#fastjsonschema.validate
+
 
     Example
     -------
@@ -127,13 +141,25 @@ def handler(event, context):
 
     if inbound_schema:
         logger.debug("Validating inbound event")
-        validate_data_against_schema(data=event, schema=inbound_schema, formats=inbound_formats)
+        validate_data_against_schema(
+            data=event,
+            schema=inbound_schema,
+            formats=inbound_formats,
+            handlers=inbound_handlers,
+            provider_options=inbound_provider_options,
+        )
 
     response = handler(event, context, **kwargs)
 
     if outbound_schema:
         logger.debug("Validating outbound event")
-        validate_data_against_schema(data=response, schema=outbound_schema, formats=outbound_formats)
+        validate_data_against_schema(
+            data=response,
+            schema=outbound_schema,
+            formats=outbound_formats,
+            handlers=outbound_handlers,
+            provider_options=outbound_provider_options,
+        )
 
     return response
 
@@ -142,6 +168,8 @@ def validate(
     event: Any,
     schema: Dict,
     formats: Optional[Dict] = None,
+    handlers: Optional[Dict] = None,
+    provider_options: Optional[Dict] = None,
     envelope: Optional[str] = None,
     jmespath_options: Optional[Dict] = None,
 ):
@@ -161,6 +189,10 @@ def validate(
         Alternative JMESPath options to be included when filtering expr
     formats: Dict
         Custom formats containing a key (e.g. int64) and a value expressed as regex or callback returning bool
+    handlers: Dict
+        Custom methods to retrieve remote schemes, keyed off of URI scheme
+    provider_options: Dict
+        Arguments that will be passed directly to the underlying validate call
 
     Example
     -------
@@ -229,4 +261,10 @@ def handler(event, context):
             jmespath_options=jmespath_options,
         )
 
-    validate_data_against_schema(data=event, schema=schema, formats=formats)
+    validate_data_against_schema(
+        data=event,
+        schema=schema,
+        formats=formats,
+        handlers=handlers,
+        provider_options=provider_options,
+    )

docs/Dockerfile

@@ -1,5 +1,5 @@
 # v9.1.18
-FROM squidfunk/mkdocs-material@sha256:96abcbb1bb6d4d79a6a142fe88150b3862b2b297fe45e6656b154d978b1787eb
+FROM squidfunk/mkdocs-material@sha256:257eca88da7f42242cd05e8cebf6d10ebd079edc207b089ad3f4f1ad107b0348
 # pip-compile --generate-hashes --output-file=requirements.txt requirements.in
 COPY requirements.txt /tmp/
 RUN pip install --require-hashes -r /tmp/requirements.txt

docs/index.md

@@ -398,7 +398,7 @@ There are many ways you can help us gain future investments to improve everyone'
 
     Add your company name and logo on our [landing page](https://powertools.aws.dev).
 
-    [:octicons-arrow-right-24: GitHub Issue template]((https://github.com/aws-powertools/powertools-lambda-python/issues/new?assignees=&labels=customer-reference&template=support_powertools.yml&title=%5BSupport+Lambda+Powertools%5D%3A+%3Cyour+organization+name%3E){target="_blank"})
+    [:octicons-arrow-right-24: GitHub Issue template](https://github.com/aws-powertools/powertools-lambda-python/issues/new?assignees=&labels=customer-reference&template=support_powertools.yml&title=%5BSupport+Lambda+Powertools%5D%3A+%3Cyour+organization+name%3E){target="_blank"}
 
 - :mega:{ .lg .middle } __Share your work__
 

docs/utilities/validation.md

@@ -147,10 +147,10 @@ Here is a handy table with built-in envelopes along with their JMESPath expressi
 | **`API_GATEWAY_HTTP`**            | `powertools_json(body)`                                       |
 | **`API_GATEWAY_REST`**            | `powertools_json(body)`                                       |
 | **`CLOUDWATCH_EVENTS_SCHEDULED`** | `detail`                                                      |
-| **`CLOUDWATCH_LOGS`**             | `awslogs.powertools_base64_gzip(data)                         | powertools_json(@).logEvents[*]` |
+| **`CLOUDWATCH_LOGS`**             | `awslogs.powertools_base64_gzip(data)` or `powertools_json(@).logEvents[*]` |
 | **`EVENTBRIDGE`**                 | `detail`                                                      |
 | **`KINESIS_DATA_STREAM`**         | `Records[*].kinesis.powertools_json(powertools_base64(data))` |
-| **`SNS`**                         | `Records[0].Sns.Message                                       | powertools_json(@)`              |
+| **`SNS`**                         | `Records[0].Sns.Message` or `powertools_json(@)`              |
 | **`SQS`**                         | `Records[*].powertools_json(body)`                            |
 
 ## Advanced
@@ -199,3 +199,33 @@ You can use our built-in [JMESPath functions](./jmespath_functions.md){target="_
 
 ???+ info
     We use these for [built-in envelopes](#built-in-envelopes) to easily to decode and unwrap events from sources like Kinesis, CloudWatch Logs, etc.
+
+### Validating with external references
+
+JSON Schema [allows schemas to reference other schemas](https://json-schema.org/understanding-json-schema/structuring#dollarref) using the `$ref` keyword with a URI value. By default, `fastjsonschema` will make a HTTP request to resolve this URI.
+
+You can use `handlers` parameter to have full control over how references schemas are fetched. This is useful when you might want to optimize caching, reducing HTTP calls, or fetching them from non-HTTP endpoints.
+
+=== "custom_handlers.py"
+
+	```python hl_lines="1 7 8 11"
+    --8<-- "examples/validation/src/custom_handlers.py"
+	```
+
+=== "custom_handlers_parent_schema"
+
+	```python hl_lines="1 7"
+    --8<-- "examples/validation/src/custom_handlers_schema.py"
+	```
+
+=== "custom_handlers_child_schema"
+
+    ```python hl_lines="12"
+    --8<-- "examples/validation/src/custom_handlers_schema.py"
+    ```
+
+=== "custom_handlers_payload.json"
+
+    ```json hl_lines="2"
+    --8<-- "examples/validation/src/custom_handlers_payload.json"
+    ```