From e984a64c35fc51856c6e9db3e334bdcdb06fb825 Mon Sep 17 00:00:00 2001 From: Leandro Damascena Date: Tue, 20 Aug 2024 17:50:36 +0100 Subject: [PATCH 1/5] Upgrade guide for v3 --- docs/upgrade.md | 278 ++++++++++++++++++++++++------------------------ 1 file changed, 137 insertions(+), 141 deletions(-) diff --git a/docs/upgrade.md b/docs/upgrade.md index 11d8cdbe83a..9ac115b804c 100644 --- a/docs/upgrade.md +++ b/docs/upgrade.md @@ -5,222 +5,218 @@ description: Guide to update between major Powertools for AWS Lambda (Python) ve -## End of support v1 +## Migrate to v3 from v2 -!!! warning "On March 31st, 2023, Powertools for AWS Lambda (Python) v1 reached end of support and will no longer receive updates or releases. If you are still using v1, we strongly recommend you to read our upgrade guide and update to the latest version." - -Given our commitment to all of our customers using Powertools for AWS Lambda (Python), we will keep [Pypi](https://pypi.org/project/aws-lambda-powertools/){target="_blank"} v1 releases and documentation 1.x versions to prevent any disruption. - -## Migrate to v2 from v1 - -We've made minimal breaking changes to make your transition to v2 as smooth as possible. +We've made minimal breaking changes to make your transition to v3 as smooth as possible. ### Quick summary -| Area | Change | Code change required | IAM Permissions change required | -| ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------- | ------------------------------- | -| **Batch** | Removed legacy [SQS batch processor](#legacy-sqs-batch-processor) in favour of **`BatchProcessor`**. | Yes | - | -| **Environment variables** | Removed legacy **`POWERTOOLS_EVENT_HANDLER_DEBUG`** in favour of [`POWERTOOLS_DEV`](index.md#optimizing-for-non-production-environments){target="_blank"}. | - | - | -| **Event Handler** | Updated [headers response format](#event-handler-headers-response-format) due to [multi-value headers and cookie support](./core/event_handler/api_gateway.md#fine-grained-responses){target="_blank"}. | Tests only | - | -| **Event Source Data Classes** | Replaced [DynamoDBStreamEvent](#dynamodbstreamevent-in-event-source-data-classes) `AttributeValue` with native Python types. | Yes | - | -| **Feature Flags** / **Parameters** | Updated [AppConfig API calls](#feature-flags-and-appconfig-parameter-utility) due to **`GetConfiguration`** API deprecation. | - | Yes | -| **Idempotency** | Updated [partition key](#idempotency-partition-key-format) to include fully qualified function/method names. | - | - | +| Area | Change | Code change required | +| ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | -------------------- | +| **Pydantic** | Removed support for [Pydantic v1](#drop-support-for-pydantic-v1) | No | +| **Parser** | Replaced [DynamoDBStreamModel](#dynamodbstreammodel-in-parser) `AttributeValue` with native Python types | Yes | +| **Lambda layer** | [Lambda layers](#new-lambda-layers-arn) are now compiled according to the specific Python version and architecture | No | +| **Batch Processor** | `@batch_processor` and `@async_batch_processor` decorators [are now deprecated](#deprecated-batch-processing-decorators) | Yes | +| **Event Source Data Classes** | New default values for optional fields | Yes | +| **Parameters** | The [default cache TTL](#parameters-default-cache-ttl-updated-to-5-minutes) is now set to **5 minutes** | No | +| **Parameters** | The `config` parameter [is deprecated](#parameters-using-the-new-boto_config-parameter) in favor of `boto_config` | Yes | +| **JMESPath Functions** | The `extract_data_from_envelope` function is [deprecated in favor](#utilizing-the-new-query-function-in-jmespath-functions) of `query` | Yes | +| **Types file** | We have removed the [type imports](#importing-types-from-typing-and-typing_annotations) from the `shared/types.py` file | Yes | ### First Steps -!!! note "All dependencies are optional now. [Tracer](core/tracer.md#install){target="_blank"}, [Validation](./utilities/validation.md#install){target="_blank"}, and [Parser](./utilities/parser.md){target="_blank"} now require additional dependencies." - Before you start, we suggest making a copy of your current working project or create a new branch with git. 1. **Upgrade** Python to at least v3.8 2. **Ensure** you have the latest version via [Lambda Layer or PyPi](index.md#install){target="_blank"}. 3. **Review** the following sections to confirm whether they affect your code -## Legacy SQS Batch Processor - -We removed the deprecated `PartialSQSProcessor` class and `sqs_batch_processor` decorator. - -You can migrate to `BatchProcessor` with the following changes: +## Drop support for Pydantic v1 -1. If you use **`sqs_batch_decorator`**, change to **`batch_processor`** decorator -2. If you use **`PartialSQSProcessor`**, change to **`BatchProcessor`** -3. [Enable **`ReportBatchItemFailures`** in your Lambda Event Source](./utilities/batch.md#required-resources){target="_blank"} -4. Change your Lambda Handler to return the new response format +!!! note "No code changes required" -=== "[Before] Decorator" +As of June 30, 2024, Pydantic v1 has reached its end-of-life, and we have discontinued support for this version. We now exclusively support Pydantic v2. - ```python hl_lines="1 6" - from aws_lambda_powertools.utilities.batch import sqs_batch_processor +You don't need to make any changes related to Powertools for AWS Lambda (Python) on your end. - def record_handler(record): - return do_something_with(record["body"]) +## DynamoDBStreamModel in parser - @sqs_batch_processor(record_handler=record_handler) - def lambda_handler(event, context): - return {"statusCode": 200} - ``` +!!! info "This also applies if you're using [**DynamoDB BatchProcessor**](https://docs.powertools.aws.dev/lambda/python/latest/utilities/batch/#processing-messages-from-dynamodb){target="_blank"}." -=== "[After] Decorator" +You will now receive native Python types when accessing DynamoDB records via `Keys`, `NewImage`, and `OldImage` attributes in `DynamoDBStreamModel`. - ```python hl_lines="3 5 11 13" - import json +Previously, you'd receive a raw JSON and need to deserialize each item to the type you'd want for convenience, or to the type DynamoDB stored via `get` method. - from aws_lambda_powertools.utilities.batch import BatchProcessor, EventType, batch_processor +With this change, you can access data deserialized as stored in DynamoDB, and no longer need to recursively deserialize nested objects (Maps) if you had them. - processor = BatchProcessor(event_type=EventType.SQS) +???+ note + For a lossless conversion of DynamoDB `Number` type, we follow AWS Python SDK (boto3) approach and convert to `Decimal`. +```python hl_lines="21-27 30-31" +from __future__ import annotations - def record_handler(record): - return do_something_with(record["body"]) +import json +from typing import Any - @batch_processor(record_handler=record_handler, processor=processor) - def lambda_handler(event, context): - return processor.response() - ``` +from aws_lambda_powertools.utilities.parser import event_parser +from aws_lambda_powertools.utilities.parser.models import DynamoDBStreamModel +from aws_lambda_powertools.utilities.typing import LambdaContext -=== "[Before] Context manager" - ```python hl_lines="1-2 4 14 19" - from aws_lambda_powertools.utilities.batch import PartialSQSProcessor - from botocore.config import Config +def send_to_sqs(data: dict): + body = json.dumps(data) + ... - config = Config(region_name="us-east-1") +@event_parser +def lambda_handler(event: DynamoDBStreamModel, context: LambdaContext): - def record_handler(record): - return_value = do_something_with(record["body"]) - return return_value + for record in event.Records: + # BEFORE - V2 + new_image: dict[str, Any] = record.dynamodb.NewImage + event_type = new_image["eventType"]["S"] + if event_type == "PENDING": + # deserialize attribute value into Python native type + # NOTE: nested objects would need additional logic + data = dict(new_image) + send_to_sqs(data) - def lambda_handler(event, context): - records = event["Records"] + # NOW - V3 + new_image: dict[str, Any] = record.dynamodb.NewImage + if new_image.get("eventType") == "PENDING": + send_to_sqs(new_image) # Here new_image is just a Python Dict type - processor = PartialSQSProcessor(config=config) +``` - with processor(records, record_handler): - result = processor.process() +## New Lambda layers ARN - return result - ``` +!!! note "No code changes required" -=== "[After] Context manager" +Our Lambda layers are now compiled according to the specific Python version and architecture, resulting in a change to the ARN. - ```python hl_lines="1 11 16" - from aws_lambda_powertools.utilities.batch import BatchProcessor, EventType, batch_processor +You need to update your deployment to include this new ARN and use the V3. Additionally, we are now including Pydantic v2 and the AWS Encryption SDK in our Lambda layers. +Check the new formats: - def record_handler(record): - return_value = do_something_with(record["body"]) - return return_value +| Layer ARN | Python version | Architecture | +| ------------------------------------------------------------------------------------------------ | --------------- | ------------ | +| arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python38-x86:{version} | 3.8 | x86_64 | +| arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python39-x86:{version} | 3.9 | x86_64 | +| arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python310-x86:{version} | 3.10 | x86_64 | +| arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python311-x86:{version} | 3.11 | x86_64 | +| arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python312-x86:{version} | 3.12 | x86_64 | +| arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python38-arm64:{version} | 3.8 | arm64 | +| arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python39-arm64:{version} | 3.9 | arm64 | +| arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python310-arm64:{version} | 3.10 | arm64 | +| arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python311-arm64:{version} | 3.11 | arm64 | +| arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python312-arm64:{version} | 3.12 | arm64 | - def lambda_handler(event, context): - records = event["Records"] +## Deprecated Batch Processing decorators - processor = BatchProcessor(event_type=EventType.SQS) +In V2, we designated `@batch_processor` and `@async_batch_processor` as legacy modes for using the Batch Processing utility. - with processor(records, record_handler): - result = processor.process() +In V3, these have been marked as deprecated. Continuing to use them will result in warnings in your IDE and during Lambda execution. - return processor.response() - ``` +```python hl_lines="17-19 22-28" +import json -## Event Handler headers response format +from aws_lambda_powertools.utilities.batch import BatchProcessor, EventType, batch_processor, process_partial_response +from aws_lambda_powertools.utilities.data_classes.sqs_event import SQSRecord +from aws_lambda_powertools.utilities.typing import LambdaContext -!!! note "No code changes required" +processor = BatchProcessor(event_type=EventType.SQS) -This only applies if you're using `APIGatewayRestResolver` and asserting custom header values in your tests. +@tracer.capture_method +def record_handler(record: SQSRecord): + payload: str = record.body + if payload: + item: dict = json.loads(payload) + logger.info(item) -Previously, custom headers were available under `headers` key in the Event Handler response. +# BEFORE - V2 +@batch_processor(record_handler=record_handler, processor=processor) +def lambda_handler(event, context: LambdaContext): + return processor.response() -```python title="V1 response headers" hl_lines="2" -{ - "headers": { - "Content-Type": "application/json" - } -} +# NOW - V3 +def lambda_handler(event, context: LambdaContext): + return process_partial_response( + event=event, + record_handler=record_handler, + processor=processor, + context=context, + ) ``` -In V2, we add all headers under `multiValueHeaders` key. This enables seamless support for multi-value headers and cookies in [fine grained responses](./core/event_handler/api_gateway.md#fine-grained-responses){target="_blank"}. +## Parameters: default cache TTL updated to 5 minutes -```python title="V2 response headers" hl_lines="2" -{ - "multiValueHeaders": { - "Content-Type": "application/json" - } -} -``` +!!! note "No code changes required" -## DynamoDBStreamEvent in Event Source Data Classes +We have updated the cache TTL to 5 minutes to reduce the number of API calls to AWS, leading to improved performance and lower costs. -!!! info "This also applies if you're using [**DynamoDB BatchProcessor**](https://docs.powertools.aws.dev/lambda/python/latest/utilities/batch/#processing-messages-from-dynamodb){target="_blank"}." +No code changes are necessary for this update; however, if you prefer the previous 5-second TTL, you can easily revert to that setting by utilizing the `max_age` parameter. -You will now receive native Python types when accessing DynamoDB records via `keys`, `new_image`, and `old_image` attributes in `DynamoDBStreamEvent`. +## Parameters: using the new boto_config parameter -Previously, you'd receive a `AttributeValue` instance and need to deserialize each item to the type you'd want for convenience, or to the type DynamoDB stored via `get_value` method. + In V2, you could use the `config` parameter to modify the **botocore Config** session settings. -With this change, you can access data deserialized as stored in DynamoDB, and no longer need to recursively deserialize nested objects (Maps) if you had them. + In V3, we renamed this parameter to `boto_config` and introduced deprecation warnings for users still utilizing `config`. -???+ note - For a lossless conversion of DynamoDB `Number` type, we follow AWS Python SDK (boto3) approach and convert to `Decimal`. +```python hl_lines="5 6 8 9" +from botocore.config import Config -```python hl_lines="15-20 24-25" -from aws_lambda_powertools.utilities.data_classes.dynamo_db_stream_event import ( - DynamoDBStreamEvent, - DynamoDBRecordEventName -) +from aws_lambda_powertools.utilities import parameters -def send_to_sqs(data: Dict): - body = json.dumps(data) - ... +# BEFORE - V2 +ssm_provider = parameters.SSMProvider(config=Config(region_name="us-west-1")) -@event_source(data_class=DynamoDBStreamEvent) -def lambda_handler(event: DynamoDBStreamEvent, context): - for record in event.records: +# NOW - V3 +ssm_provider = parameters.SSMProvider(boto_config=Config(region_name="us-west-1")) - # BEFORE - new_image: Dict[str, AttributeValue] = record.dynamodb.new_image - event_type: AttributeValue = new_image["eventType"].get_value - if event_type == "PENDING": - # deserialize attribute value into Python native type - # NOTE: nested objects would need additional logic - data = {k: v.get_value for k, v in image.items()} - send_to_sqs(data) - - # AFTER - new_image: Dict[str, Any] = record.dynamodb.new_image - if new_image.get("eventType") == "PENDING": - send_to_sqs(new_image) # Here new_image is just a Python Dict type +def handler(event, context): + value = ssm_provider.get("/my/parameter") + return {"message": value} ``` -## Feature Flags and AppConfig Parameter utility +## Utilizing the new query function in JMESPath Functions -!!! note "No code changes required" +In V2, you could use the `extract_data_from_envelope` function to search and extract data from dictionaries with JMESPath. -We replaced `GetConfiguration` API ([now deprecated](https://github.com/aws-powertools/powertools-lambda-python/issues/1506#issuecomment-1266645884){target="_blank"}) with `GetLatestConfiguration` and `StartConfigurationSession`. +In V3, we renamed this function to `query` and introduced deprecation warnings for users still utilizing `extract_data_from_envelope`. -As such, you must update your IAM Role permissions to allow the following IAM actions: +```python hl_lines="6-7 9-10" +from aws_lambda_powertools.utilities.jmespath_utils import extract_data_from_envelope, query +from aws_lambda_powertools.utilities.typing import LambdaContext -* `appconfig:GetLatestConfiguration` -* `appconfig:StartConfigurationSession` -## Idempotency partition key format +def handler(event: dict, context: LambdaContext) -> dict: + # BEFORE - V2 + some_data = extract_data_from_envelope(data=event, envelope="powertools_json(body)") -!!! note "No code changes required" + # NOW - V3 + some_data = query(data=event, envelope="powertools_json(body)") -We replaced the DynamoDB partition key format to include fully qualified function/method names. This means that recent non-expired idempotent transactions will be ignored. + return {"data": some_data} +``` -Previously, we used the function/method name to generate the partition key value. +## Importing types from typing and typing_annotations -> e.g. `HelloWorldFunction.lambda_handler#99914b932bd37a50b983c5e7c90ae93b` +We refactored our codebase to align with some PEP style guidelines and eliminated the use of `aws_lambda_powertools.shared.types` imports. Instead, we now utilize types from the standard `typing` library, which are compatible with Python versions 3.8 and above, or from `typing_extensions` for additional type support. -![Idempotency Before](./media/upgrade_idempotency_before.png) +Since V2, we have included `typing_extensions` as a dependency. If you require additional types is not supported by a specific Python version, you can import them from `typing_extensions`. -In V2, we now distinguish between distinct classes or modules that may have the same function/method name. +```python hl_lines="1 2 4 5" +# BEFORE - V2 +from aws_lambda_powertools.shared.types import Annotated -[For example](https://github.com/aws-powertools/powertools-lambda-python/issues/1330){target="_blank"}, an ABC or Protocol class may have multiple implementations of `process_payment` method and may have different results. +# NOW - V3 +from typing_extensions import Annotated - +from aws_lambda_powertools.utilities.typing import LambdaContext -> e.g. `HelloWorldFunction.app.lambda_handler#99914b932bd37a50b983c5e7c90ae93b` -![Idempotency Before](./media/upgrade_idempotency_after.png) +def handler(event: dict, context: LambdaContext) -> dict: + ... + +``` From 8b6332d830bcc163a4cbef698e4ce3ade313c722 Mon Sep 17 00:00:00 2001 From: Leandro Damascena Date: Tue, 20 Aug 2024 19:13:40 +0100 Subject: [PATCH 2/5] Upgrade guide for v3 --- docs/upgrade.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/docs/upgrade.md b/docs/upgrade.md index 9ac115b804c..5318196581f 100644 --- a/docs/upgrade.md +++ b/docs/upgrade.md @@ -17,7 +17,7 @@ We've made minimal breaking changes to make your transition to v3 as smooth as p | **Parser** | Replaced [DynamoDBStreamModel](#dynamodbstreammodel-in-parser) `AttributeValue` with native Python types | Yes | | **Lambda layer** | [Lambda layers](#new-lambda-layers-arn) are now compiled according to the specific Python version and architecture | No | | **Batch Processor** | `@batch_processor` and `@async_batch_processor` decorators [are now deprecated](#deprecated-batch-processing-decorators) | Yes | -| **Event Source Data Classes** | New default values for optional fields | Yes | +| **Event Source Data Classes** | Updated [default values](#event-source-default-values) for optional fields. | Yes | | **Parameters** | The [default cache TTL](#parameters-default-cache-ttl-updated-to-5-minutes) is now set to **5 minutes** | No | | **Parameters** | The `config` parameter [is deprecated](#parameters-using-the-new-boto_config-parameter) in favor of `boto_config` | Yes | | **JMESPath Functions** | The `extract_data_from_envelope` function is [deprecated in favor](#utilizing-the-new-query-function-in-jmespath-functions) of `query` | Yes | @@ -37,7 +37,7 @@ Before you start, we suggest making a copy of your current working project or cr As of June 30, 2024, Pydantic v1 has reached its end-of-life, and we have discontinued support for this version. We now exclusively support Pydantic v2. -You don't need to make any changes related to Powertools for AWS Lambda (Python) on your end. +You don't need to make any changes to your code regarding Powertools for AWS Lambda (Python), as we fully support Pydantic V2 features. ## DynamoDBStreamModel in parser @@ -148,6 +148,14 @@ def lambda_handler(event, context: LambdaContext): ) ``` +## Event source default values + +We've modified the **Event Source Data classes** so that optional dictionaries and lists now return empty dictionaries or lists instead of None. This improvement simplifies your code by eliminating the need for conditional checks when accessing these fields, while maintaining backward compatibility with previous implementations. + +We've applied this change broadly across various event source data classes, ensuring a more consistent and streamlined coding experience for you. + +**MISSING BEFORE AND AFTER** + ## Parameters: default cache TTL updated to 5 minutes !!! note "No code changes required" From 3ee7a20273e448a21da98fb47035f95785d2284d Mon Sep 17 00:00:00 2001 From: Leandro Damascena Date: Fri, 23 Aug 2024 11:39:51 +0100 Subject: [PATCH 3/5] Adding Pydantic migration guide reference --- docs/upgrade.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/upgrade.md b/docs/upgrade.md index 5318196581f..19f6a5709a6 100644 --- a/docs/upgrade.md +++ b/docs/upgrade.md @@ -37,7 +37,7 @@ Before you start, we suggest making a copy of your current working project or cr As of June 30, 2024, Pydantic v1 has reached its end-of-life, and we have discontinued support for this version. We now exclusively support Pydantic v2. -You don't need to make any changes to your code regarding Powertools for AWS Lambda (Python), as we fully support Pydantic V2 features. +If you are defining your own Pydantic models and want to learn how to migrate them to Pydantic v2, please refer to the [Pydantic v2 Migration Guide](https://docs.pydantic.dev/latest/migration/){target="_blank"} for detailed instructions. ## DynamoDBStreamModel in parser From 15752409f8fa677a628f5544328beaa4c62e339c Mon Sep 17 00:00:00 2001 From: Leandro Damascena Date: Tue, 10 Sep 2024 23:52:19 +0100 Subject: [PATCH 4/5] Updating upg guide --- docs/upgrade.md | 225 ++++++++++++++++++++++++++++++------------------ 1 file changed, 143 insertions(+), 82 deletions(-) diff --git a/docs/upgrade.md b/docs/upgrade.md index 19f6a5709a6..036b0f20da4 100644 --- a/docs/upgrade.md +++ b/docs/upgrade.md @@ -13,23 +13,24 @@ We've made minimal breaking changes to make your transition to v3 as smooth as p | Area | Change | Code change required | | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | -------------------- | -| **Pydantic** | Removed support for [Pydantic v1](#drop-support-for-pydantic-v1) | No | -| **Parser** | Replaced [DynamoDBStreamModel](#dynamodbstreammodel-in-parser) `AttributeValue` with native Python types | Yes | -| **Lambda layer** | [Lambda layers](#new-lambda-layers-arn) are now compiled according to the specific Python version and architecture | No | +| **Pydantic** | We have removed support for [Pydantic v1](#drop-support-for-pydantic-v1) | No | +| **Parser** | We have replaced [DynamoDBStreamModel](#dynamodbstreammodel-in-parser) `AttributeValue` with native Python types | Yes | +| **Lambda layer** | [Lambda layers](#new-aws-lambda-layer-arns) are now compiled according to the specific Python version and architecture | No | +| **Event Handler** | We [have deprecated](#event-handler-headers-are-case-insensitive) the `get_header_value` function. | Yes | | **Batch Processor** | `@batch_processor` and `@async_batch_processor` decorators [are now deprecated](#deprecated-batch-processing-decorators) | Yes | -| **Event Source Data Classes** | Updated [default values](#event-source-default-values) for optional fields. | Yes | -| **Parameters** | The [default cache TTL](#parameters-default-cache-ttl-updated-to-5-minutes) is now set to **5 minutes** | No | -| **Parameters** | The `config` parameter [is deprecated](#parameters-using-the-new-boto_config-parameter) in favor of `boto_config` | Yes | -| **JMESPath Functions** | The `extract_data_from_envelope` function is [deprecated in favor](#utilizing-the-new-query-function-in-jmespath-functions) of `query` | Yes | -| **Types file** | We have removed the [type imports](#importing-types-from-typing-and-typing_annotations) from the `shared/types.py` file | Yes | +| **Event Source Data Classes** | We have updated [default values](#event-source-default-values) for optional fields. | Yes | +| **Parameters** | The [default cache TTL](#parameters-default-cache-ttl-updated-to-5-minutes) is now set to **5 minutes** | No | +| **Parameters** | The `config` parameter [is deprecated](#parameters-using-the-new-boto_config-parameter) in favor of `boto_config` | Yes | +| **JMESPath Functions** | The `extract_data_from_envelope` function is [deprecated in favor](#utilizing-the-new-query-function-in-jmespath-functions) of `query` | Yes | +| **Types file** | We have removed the [type imports](#importing-types-from-typing-and-typing_annotations) from the `shared/types.py` file | Yes | ### First Steps Before you start, we suggest making a copy of your current working project or create a new branch with git. -1. **Upgrade** Python to at least v3.8 +1. **Upgrade** Python to at least v3.8. 2. **Ensure** you have the latest version via [Lambda Layer or PyPi](index.md#install){target="_blank"}. -3. **Review** the following sections to confirm whether they affect your code +3. **Review** the following sections to confirm if you need to make changes to your code. ## Drop support for Pydantic v1 @@ -37,13 +38,13 @@ Before you start, we suggest making a copy of your current working project or cr As of June 30, 2024, Pydantic v1 has reached its end-of-life, and we have discontinued support for this version. We now exclusively support Pydantic v2. -If you are defining your own Pydantic models and want to learn how to migrate them to Pydantic v2, please refer to the [Pydantic v2 Migration Guide](https://docs.pydantic.dev/latest/migration/){target="_blank"} for detailed instructions. +Use [Pydantic v2 Migration Guide](https://docs.pydantic.dev/latest/migration/){target="_blank"} to migrate your custom Pydantic models to v2. ## DynamoDBStreamModel in parser !!! info "This also applies if you're using [**DynamoDB BatchProcessor**](https://docs.powertools.aws.dev/lambda/python/latest/utilities/batch/#processing-messages-from-dynamodb){target="_blank"}." -You will now receive native Python types when accessing DynamoDB records via `Keys`, `NewImage`, and `OldImage` attributes in `DynamoDBStreamModel`. +`DynamoDBStreamModel` now returns native Python types when you access DynamoDB records through `Keys`, `NewImage`, and `OldImage` attributes. Previously, you'd receive a raw JSON and need to deserialize each item to the type you'd want for convenience, or to the type DynamoDB stored via `get` method. @@ -52,7 +53,7 @@ With this change, you can access data deserialized as stored in DynamoDB, and no ???+ note For a lossless conversion of DynamoDB `Number` type, we follow AWS Python SDK (boto3) approach and convert to `Decimal`. -```python hl_lines="21-27 30-31" +```diff from __future__ import annotations import json @@ -72,52 +73,96 @@ def lambda_handler(event: DynamoDBStreamModel, context: LambdaContext): for record in event.Records: - # BEFORE - V2 - new_image: dict[str, Any] = record.dynamodb.NewImage - event_type = new_image["eventType"]["S"] - if event_type == "PENDING": - # deserialize attribute value into Python native type - # NOTE: nested objects would need additional logic - data = dict(new_image) - send_to_sqs(data) +- # BEFORE - v2 +- new_image: dict[str, Any] = record.dynamodb.NewImage +- event_type = new_image["eventType"]["S"] +- if event_type == "PENDING": +- # deserialize attribute value into Python native type +- # NOTE: nested objects would need additional logic +- data = dict(new_image) +- send_to_sqs(data) - # NOW - V3 - new_image: dict[str, Any] = record.dynamodb.NewImage - if new_image.get("eventType") == "PENDING": - send_to_sqs(new_image) # Here new_image is just a Python Dict type ++ # NOW - v3 ++ new_image: dict[str, Any] = record.dynamodb.NewImage ++ if new_image.get("eventType") == "PENDING": ++ send_to_sqs(new_image) # Here new_image is just a Python Dict type ``` -## New Lambda layers ARN +## New AWS Lambda Layer ARNs !!! note "No code changes required" -Our Lambda layers are now compiled according to the specific Python version and architecture, resulting in a change to the ARN. +To give you better a better experience, we're now building Powertools for AWS Lambda (Python)'s Lambda layers for specific Python versions (`3.8-3.12`) and architectures (`x86_64` & `arm64`). -You need to update your deployment to include this new ARN and use the V3. Additionally, we are now including Pydantic v2 and the AWS Encryption SDK in our Lambda layers. +This also allows us to include architecture-specific versions of both Pydantic v2 and AWS Encryption SDK and give you a more streamlined setup. -Check the new formats: +To take advantage of the new layers, you need to update your functions or deployment setup to include one of the new Lambda layer ARN from the table below: -| Layer ARN | Python version | Architecture | -| ------------------------------------------------------------------------------------------------ | --------------- | ------------ | -| arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python38-x86:{version} | 3.8 | x86_64 | -| arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python39-x86:{version} | 3.9 | x86_64 | -| arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python310-x86:{version} | 3.10 | x86_64 | -| arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python311-x86:{version} | 3.11 | x86_64 | -| arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python312-x86:{version} | 3.12 | x86_64 | -| arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python38-arm64:{version} | 3.8 | arm64 | -| arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python39-arm64:{version} | 3.9 | arm64 | -| arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python310-arm64:{version} | 3.10 | arm64 | -| arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python311-arm64:{version} | 3.11 | arm64 | -| arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python312-arm64:{version} | 3.12 | arm64 | +| Architecture | Python version | Layer ARN | +| ------------ | -------------- | ------------------------------------------------------------------------------------------------ | +| x86_64 | 3.8 | arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python38-x86:{version} | +| x86_64 | 3.9 | arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python39-x86:{version} | +| x86_64 | 3.10 | arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python310-x86:{version} | +| x86_64 | 3.11 | arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python311-x86:{version} | +| x86_64 | 3.12 | arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python312-x86:{version} | +| arm64 | 3.8 | arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python38-arm64:{version} | +| arm64 | 3.9 | arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python39-arm64:{version} | +| arm64 | 3.10 | arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python310-arm64:{version} | +| arm64 | 3.11 | arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python311-arm64:{version} | +| arm64 | 3.12 | arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV3-python312-arm64:{version} | + +## Event Handler: headers are case-insensitive + +According to the [HTTP RFC](https://datatracker.ietf.org/doc/html/rfc9110#section-5.1){target="_blank"}, HTTP headers are case-insensitive. As a result, we have deprecated the `get_header_value` function to align with this standard. Instead, we recommend using `app.current_event.headers.get` to access header values directly + +Consequently, the `case_sensitive` parameter in this function no longer has any effect, as we now ensure consistent casing by normalizing headers for you. This function will be removed in a future release, and we encourage users to adopt the new method to access header values. + +```diff +import requests +from requests import Response + +from aws_lambda_powertools import Logger, Tracer +from aws_lambda_powertools.event_handler import APIGatewayRestResolver +from aws_lambda_powertools.logging import correlation_paths +from aws_lambda_powertools.utilities.typing import LambdaContext + +tracer = Tracer() +logger = Logger() +app = APIGatewayRestResolver() + + +@app.get("/todos") +@tracer.capture_method +def get_todos(): + endpoint = "https://jsonplaceholder.typicode.com/todos" + + # BEFORE - v2 +- api_key: str = app.current_event.get_header_value(name="X-Api-Key", case_sensitive=True, default_value="") + + # NOW - v3 ++ api_key: str = app.current_event.headers.get("X-Api-Key", "") + + todos: Response = requests.get(endpoint, headers={"X-Api-Key": api_key}) + todos.raise_for_status() + + return {"todos": todos.json()} + + +# You can continue to use other utilities just as before +@logger.inject_lambda_context(correlation_id_path=correlation_paths.API_GATEWAY_REST) +@tracer.capture_lambda_handler +def lambda_handler(event: dict, context: LambdaContext) -> dict: + return app.resolve(event, context) +``` ## Deprecated Batch Processing decorators -In V2, we designated `@batch_processor` and `@async_batch_processor` as legacy modes for using the Batch Processing utility. +In v2, we designated `@batch_processor` and `@async_batch_processor` as legacy modes for using the Batch Processing utility. -In V3, these have been marked as deprecated. Continuing to use them will result in warnings in your IDE and during Lambda execution. +In v3, these have been marked as deprecated. Continuing to use them will result in warnings in your IDE and during Lambda execution. -```python hl_lines="17-19 22-28" +```diff import json from aws_lambda_powertools.utilities.batch import BatchProcessor, EventType, batch_processor, process_partial_response @@ -133,53 +178,69 @@ def record_handler(record: SQSRecord): item: dict = json.loads(payload) logger.info(item) -# BEFORE - V2 -@batch_processor(record_handler=record_handler, processor=processor) -def lambda_handler(event, context: LambdaContext): - return processor.response() - -# NOW - V3 -def lambda_handler(event, context: LambdaContext): - return process_partial_response( - event=event, - record_handler=record_handler, - processor=processor, - context=context, - ) +-# BEFORE - v2 +-@batch_processor(record_handler=record_handler, processor=processor) +-def lambda_handler(event, context: LambdaContext): +- return processor.response() + ++ # NOW - v3 ++def lambda_handler(event, context: LambdaContext): ++ return process_partial_response( ++ event=event, ++ record_handler=record_handler, ++ processor=processor, ++ context=context, ++ ) ``` ## Event source default values -We've modified the **Event Source Data classes** so that optional dictionaries and lists now return empty dictionaries or lists instead of None. This improvement simplifies your code by eliminating the need for conditional checks when accessing these fields, while maintaining backward compatibility with previous implementations. +We've modified the **Event Source Data classes** so that optional dictionaries and lists now return empty strings, dictionaries or lists instead of `None`. This improvement simplifies your code by eliminating the need for conditional checks when accessing these fields, while maintaining backward compatibility with previous implementations. We've applied this change broadly across various event source data classes, ensuring a more consistent and streamlined coding experience for you. -**MISSING BEFORE AND AFTER** +```diff +from aws_lambda_powertools.utilities.data_classes import DynamoDBStreamEvent, event_source +from aws_lambda_powertools.utilities.typing import LambdaContext + + +@event_source(data_class=DynamoDBStreamEvent) +def lambda_handler(event: DynamoDBStreamEvent, context: LambdaContext): + for record in event.records: + +- # BEFORE - v2 +- old_image_type_return_v2 = type(record.dynamodb.old_image) +- # Output is + ++ # NOW - v3 ++ old_image_type_return_v3 = type(record.dynamodb.old_image) ++ # Output is +``` ## Parameters: default cache TTL updated to 5 minutes !!! note "No code changes required" -We have updated the cache TTL to 5 minutes to reduce the number of API calls to AWS, leading to improved performance and lower costs. +We have updated the cache TTL from 5 seconds to 5 minutes to reduce the number of API calls to AWS, leading to improved performance and lower costs. -No code changes are necessary for this update; however, if you prefer the previous 5-second TTL, you can easily revert to that setting by utilizing the `max_age` parameter. +No code changes are necessary for this update; however, if you prefer the previous behavior, you can set the `max_age` parameter back to 5 seconds. ## Parameters: using the new boto_config parameter - In V2, you could use the `config` parameter to modify the **botocore Config** session settings. +In v2, you could use the `config` parameter to modify the **botocore Config** session settings. - In V3, we renamed this parameter to `boto_config` and introduced deprecation warnings for users still utilizing `config`. +In v3, we renamed this parameter to `boto_config` to standardize the name with other features, such as Idempotency, and introduced deprecation warnings for users still using `config`. -```python hl_lines="5 6 8 9" +```diff from botocore.config import Config from aws_lambda_powertools.utilities import parameters -# BEFORE - V2 -ssm_provider = parameters.SSMProvider(config=Config(region_name="us-west-1")) +-# BEFORE - v2 +-ssm_provider = parameters.SSMProvider(config=Config(region_name="us-west-1")) -# NOW - V3 -ssm_provider = parameters.SSMProvider(boto_config=Config(region_name="us-west-1")) ++# NOW - v3 ++ssm_provider = parameters.SSMProvider(boto_config=Config(region_name="us-west-1")) def handler(event, context): value = ssm_provider.get("/my/parameter") @@ -189,37 +250,37 @@ def handler(event, context): ## Utilizing the new query function in JMESPath Functions -In V2, you could use the `extract_data_from_envelope` function to search and extract data from dictionaries with JMESPath. +In v2, you could use the `extract_data_from_envelope` function to search and extract data from dictionaries with JMESPath. This name was too generic and customers told us it was confusing. -In V3, we renamed this function to `query` and introduced deprecation warnings for users still utilizing `extract_data_from_envelope`. +In v3, we renamed this function to `query` to align with similar frameworks in the ecosystem, and introduced deprecation warnings for users still using `extract_data_from_envelope`. -```python hl_lines="6-7 9-10" +```diff from aws_lambda_powertools.utilities.jmespath_utils import extract_data_from_envelope, query from aws_lambda_powertools.utilities.typing import LambdaContext def handler(event: dict, context: LambdaContext) -> dict: - # BEFORE - V2 - some_data = extract_data_from_envelope(data=event, envelope="powertools_json(body)") +- # BEFORE - v2 +- some_data = extract_data_from_envelope(data=event, envelope="powertools_json(body)") - # NOW - V3 - some_data = query(data=event, envelope="powertools_json(body)") ++ # NOW - v3 ++ some_data = query(data=event, envelope="powertools_json(body)") return {"data": some_data} ``` ## Importing types from typing and typing_annotations -We refactored our codebase to align with some PEP style guidelines and eliminated the use of `aws_lambda_powertools.shared.types` imports. Instead, we now utilize types from the standard `typing` library, which are compatible with Python versions 3.8 and above, or from `typing_extensions` for additional type support. +We refactored our codebase to align with Python guidelines and eliminated the use of `aws_lambda_powertools.shared.types` imports. -Since V2, we have included `typing_extensions` as a dependency. If you require additional types is not supported by a specific Python version, you can import them from `typing_extensions`. +Instead, we now utilize types from the standard `typing` library, which are compatible with Python versions 3.8 and above, or from `typing_extensions` (included as a required dependency) for additional type support. -```python hl_lines="1 2 4 5" -# BEFORE - V2 -from aws_lambda_powertools.shared.types import Annotated +```diff +-# BEFORE - v2 +-from aws_lambda_powertools.shared.types import Annotated -# NOW - V3 -from typing_extensions import Annotated ++# NOW - v3 ++from typing_extensions import Annotated from aws_lambda_powertools.utilities.typing import LambdaContext From 6c9fff21a05b4bd8e378bc41cc63fed262fc726b Mon Sep 17 00:00:00 2001 From: Leandro Damascena Date: Thu, 12 Sep 2024 14:26:04 +0100 Subject: [PATCH 5/5] Upgrade guide update --- docs/upgrade.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/upgrade.md b/docs/upgrade.md index 036b0f20da4..9d29be758e6 100644 --- a/docs/upgrade.md +++ b/docs/upgrade.md @@ -7,6 +7,8 @@ description: Guide to update between major Powertools for AWS Lambda (Python) ve ## Migrate to v3 from v2 +!!! info "We strongly encourage you to migrate to v3. However, if you still need to upgrade from v1 to v2, you can find the [upgrade guide](/lambda/python/2.43.1/upgrade/)." + We've made minimal breaking changes to make your transition to v3 as smooth as possible. ### Quick summary