diff --git a/docs/utilities/idempotency.md b/docs/utilities/idempotency.md index 12baabd047e..06bf15748cb 100644 --- a/docs/utilities/idempotency.md +++ b/docs/utilities/idempotency.md @@ -5,26 +5,26 @@ description: Utility -The idempotency utility provides a simple solution to convert your Lambda functions into idempotent operations which are safe to retry. +The idempotency utility allows you to retry operations within a time window with the same input, producing the same output. ## Key features -* Prevent Lambda handler from executing more than once on the same event payload during a time window -* Ensure Lambda handler returns the same result when called with the same payload -* Select a subset of the event as the idempotency key using JMESPath expressions -* Set a time window in which records with the same payload should be considered duplicates -* Expires in-progress executions if the Lambda function times out halfway through -* Support Amazon DynamoDB and Redis as persistence layers +* Produces the previous successful result when a function is called repeatedly with the same idempotency key +* Choose your idempotency key from one or more fields, or entire payload +* Safeguard concurrent requests, timeouts, missing idempotency keys, and payload tampering +* Support for Amazon DynamoDB, Redis, bring your own persistence layer, and in-memory caching ## Terminology The property of idempotency means that an operation does not cause additional side effects if it is called more than once with the same input parameters. -**Idempotent operations will return the same result when they are called multiple times with the same parameters**. This makes idempotent operations safe to retry. +**Idempotency key** is a combination of **(a)** Lambda function name, **(b)** fully qualified name of your function, and **(c)** a hash of the entire payload or part(s) of the payload you specify. -**Idempotency key** is a hash representation of either the entire event or a specific configured subset of the event, and invocation results are **JSON serialized** and stored in your persistence storage layer. +**Idempotent request** is an operation with the same input previously processed that is not expired in your persistent storage or in-memory cache. -**Idempotency record** is the data representation of an idempotent request saved in your preferred storage layer. We use it to coordinate whether a request is idempotent, whether it's still valid or expired based on timestamps, etc. +**Persistence layer** is a storage we use to create, read, expire, and delete idempotency records. + +**Idempotency record** is the data representation of an idempotent request saved in the persistent layer and in its various status. We use it to coordinate whether **(a)** a request is idempotent, **(b)** it's not expired, **(c)** JSON response to return, and more.
```mermaid @@ -35,7 +35,7 @@ classDiagram status Status expiry_timestamp int in_progress_expiry_timestamp int - response_data Json~str~ + response_data str~JSON~ payload_hash str } class Status { @@ -52,33 +52,64 @@ classDiagram ## Getting started -???+ note - This section uses DynamoDB as the default idempotent persistence storage layer. If you are interested in using Redis as the persistence storage layer, check out the [Redis as persistence storage layer](#redis-as-persistent-storage-layer-provider) Section. +We use Amazon DynamoDB as the default persistence layer in the documentation. If you prefer Redis, you can learn more from [this section](#redis-database). ### IAM Permissions -Your Lambda function IAM Role must have `dynamodb:GetItem`, `dynamodb:PutItem`, `dynamodb:UpdateItem` and `dynamodb:DeleteItem` IAM permissions before using this feature. +When using Amazon DynamoDB as the persistence layer, you will need the following IAM permissions: -???+ note - If you're using our example [AWS Serverless Application Model (SAM)](#required-resources), [AWS Cloud Development Kit (CDK)](#required-resources), or [Terraform](#required-resources) it already adds the required permissions. +| IAM Permission | Operation | +| ------------------------------------ | ------------------------------------------------------------------------ | +| **`dynamodb:GetItem`**{: .copyMe} | Retrieve idempotent record _(strong consistency)_ | +| **`dynamodb:PutItem`**{: .copyMe} | New idempotent records, replace expired idempotent records | +| **`dynamodb:UpdateItem`**{: .copyMe} | Complete idempotency transaction, and/or update idempotent records state | +| **`dynamodb:DeleteItem`**{: .copyMe} | Delete idempotent records for unsuccessful idempotency transactions | + +**First time setting it up?** + +We provide Infrastrucure as Code examples with [AWS Serverless Application Model (SAM)](#aws-serverless-application-model-sam-example), [AWS Cloud Development Kit (CDK)](#aws-cloud-development-kit-cdk), and [Terraform](#terraform) with the required permissions. ### Required resources -Before getting started, you need to create a persistent storage layer where the idempotency utility can store its state - your lambda functions will need read and write access to it. +To start, you'll need: + + + +
+* :octicons-database-16:{ .lg .middle } __Persistent storage__ + + --- + + [Amazon DynamoDB](#dynamodb-table) or [Redis](#redis-database) + +* :simple-awslambda:{ .lg .middle } **AWS Lambda function** + + --- -We currently support Amazon DynamoDB and Redis as a storage layer. The following example demonstrates how to create a table in DynamoDB. If you prefer to use Redis, refer go to the section [RedisPersistenceLayer](#redispersistencelayer) section. + With permissions to use your persistent storage -**Default table configuration** +
-If you're not [changing the default configuration for the DynamoDB persistence layer](#dynamodbpersistencelayer), this is the expected default configuration: + -| Configuration | Value | Notes | -| ------------------ | ------------ |-------------------------------------------------------------------------------------| -| Partition key | `id` | | -| TTL attribute name | `expiration` | This can only be configured after your table is created if you're using AWS Console | +!!! note "Primary key for any persistence storage" + We combine the Lambda function name and the [fully qualified name](https://peps.python.org/pep-3155/){target="_blank" rel="nofollow"} for classes/functions to + prevent accidental reuse for similar code sharing input/output. -???+ tip "Tip: You can share a single state table for all functions" - You can reuse the same DynamoDB table to store idempotency state. We add `module_name` and [qualified name for classes and functions](https://peps.python.org/pep-3155/){target="_blank" rel="nofollow"} in addition to the idempotency key as a hash key. + Primary key sample: `{lambda_fn_name}.{module_name}.{fn_qualified_name}#{idempotency_key_hash}` + +#### DynamoDB table + +Unless you're looking to use an [existing table or customize each attribute](#dynamodbpersistencelayer), you only need the following: + +| Configuration | Value | Notes | +| ------------------ | ------------ | ------------------------------------------------------------ | +| Partition key | `id` | | +| TTL attribute name | `expiration` | Using AWS Console? This is configurable after table creation | + +You **can** use a single DynamoDB table for all functions annotated with Idempotency. + +##### DynamoDB IaC examples === "AWS Serverless Application Model (SAM) example" @@ -96,67 +127,82 @@ If you're not [changing the default configuration for the DynamoDB persistence l ```terraform hl_lines="14-26 64-70" --8<-- "examples/idempotency/templates/terraform.tf" ``` +` -???+ warning "Warning: Large responses with DynamoDB persistence layer" - When using this utility with DynamoDB, your function's responses must be [smaller than 400KB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html#limits-items){target="_blank"}. +##### Limitations - Larger items cannot be written to DynamoDB and will cause exceptions. If your response exceeds 400kb, consider using Redis as your persistence layer. +* **DynamoDB restricts [item sizes to 400KB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html#limits-items){target="_blank"}**. This means that if your annotated function's response must be smaller than 400KB, otherwise your function will fail. Consider [Redis](#redis-database) as an alternative. - -???+ info "Info: DynamoDB" +* **Expect 2 WCU per non-idempotent call**. During the first invocation, we use `PutItem` for locking and `UpdateItem` for completion. Consider reviewing [DynamoDB pricing documentation](https://aws.amazon.com/dynamodb/pricing/){target="_blank"} to estimate cost. - During the first invocation with a payload, the Lambda function executes both a `PutItem` and an `UpdateItem` operations to store the data in DynamoDB. If the result returned by your Lambda is less than 1kb, you can expect 2 WCUs per Lambda invocation. +* **Old boto3 versions can increase costs**. For cost optimization, we use a conditional `PutItem` to always lock a new idempotency record. If locking fails, it means we already have an idempotency record saving us an additional `GetItem` call. However, this is only supported in boto3 `1.26.194` and higher _([June 30th 2023](https://aws.amazon.com/about-aws/whats-new/2023/06/amazon-dynamodb-cost-failed-conditional-writes/){target="_blank"})_. - On subsequent invocations with the same payload, you can expect just 1 `PutItem` request to DynamoDB. +#### Redis database + +We recommend you start with a Redis compatible management services such as [Amazon ElastiCache for Redis](https://aws.amazon.com/elasticache/redis/){target="_blank"} or [Amazon MemoryDB for Redis](https://aws.amazon.com/memorydb/){target="_blank"}. + +In both services, you'll need to configure [VPC access](https://docs.aws.amazon.com/lambda/latest/dg/configuration-vpc.html){target="_blank"} to your AWS Lambda. + +##### Redis IaC examples + +=== "AWS CloudFormation example" - **Note:** While we try to minimize requests to DynamoDB to 1 per invocation, if your boto3 version is lower than `1.26.194`, you may experience 2 requests in every invocation. Ensure to check your boto3 version and review the [DynamoDB pricing documentation](https://aws.amazon.com/dynamodb/pricing/){target="_blank"} to estimate the cost. + !!! tip inline end "Prefer AWS Console/CLI?" + + Follow the official tutorials for [Amazon ElastiCache for Redis](https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/LambdaRedis.html) or [Amazon MemoryDB for Redis](https://aws.amazon.com/blogs/database/access-amazon-memorydb-for-redis-from-aws-lambda/) + + ```yaml hl_lines="5 21" + --8<-- "examples/idempotency/templates/cfn_redis_serverless.yaml" + ``` + + 1. Replace the Security Group ID and Subnet ID to match your VPC settings. + 2. Replace the Security Group ID and Subnet ID to match your VPC settings. + +Once setup, you can find a quick start and advanced examples for Redis in [the persistent layers section](#redispersistencelayer). -### Idempotent decorator -You can quickly start by initializing the `DynamoDBPersistenceLayer` class and using it with the `idempotent` decorator on your lambda handler. +### Idempotent decorator -???+ note - In this example, the entire Lambda handler is treated as a single idempotent operation. If your Lambda handler can cause multiple side effects, or you're only interested in making a specific logic idempotent, use [`idempotent_function`](#idempotent_function-decorator) instead. +For simple use cases, you can use the `idempotent` decorator on your Lambda handler function. -!!! tip "See [Choosing a payload subset for idempotency](#choosing-a-payload-subset-for-idempotency) for more elaborate use cases." +It will treat the entire event as an idempotency key. That is, the same event will return the previously stored result within a [configurable time window](#adjusting-expiration-window) _(1 hour, by default)_. === "Idempotent decorator" - ```python hl_lines="4-7 10 24" + !!! tip "You can also choose [one or more fields](#choosing-a-payload-subset) as an idempotency key." + + ```python title="getting_started_with_idempotency.py" hl_lines="5-8 12 25" --8<-- "examples/idempotency/src/getting_started_with_idempotency.py" ``` === "Sample event" - ```json + ```json title="getting_started_with_idempotency_payload.json" --8<-- "examples/idempotency/src/getting_started_with_idempotency_payload.json" ``` -After processing this request successfully, a second request containing the exact same payload above will now return the same response, ensuring our customer isn't charged twice. - -!!! question "New to idempotency concept? Please review our [Terminology](#terminology) section if you haven't yet." - ### Idempotent_function decorator -Similar to [idempotent decorator](#idempotent-decorator), you can use `idempotent_function` decorator for any synchronous Python function. - -When using `idempotent_function`, you must tell us which keyword parameter in your function signature has the data we should use via **`data_keyword_argument`**. +For full flexibility, you can use the `idempotent_function` decorator for any synchronous Python function. -!!! tip "We support JSON serializable data, [Python Dataclasses](https://docs.python.org/3.12/library/dataclasses.html){target="_blank" rel="nofollow"}, [Parser/Pydantic Models](parser.md){target="_blank"}, and our [Event Source Data Classes](./data_classes.md){target="_blank"}." +When using this decorator, you **must** call your decorated function using keyword arguments. -???+ warning "Limitation" - Make sure to call your decorated function using keyword arguments. +You can use `data_keyword_argument` to tell us the argument to extract an idempotency key. We support JSON serializable data, [Dataclasses](https://docs.python.org/3.12/library/dataclasses.html){target="_blank" rel="nofollow"}, Pydantic Models, and [Event Source Data Classes](./data_classes.md){target="_blank"} === "Using Dataclasses" - ```python hl_lines="3-7 11 26 37" + ```python title="working_with_idempotent_function_dataclass.py" hl_lines="4-8 12 28 41" --8<-- "examples/idempotency/src/working_with_idempotent_function_dataclass.py" ``` + 1. Notice how **`data_keyword_argument`** matches the name of the parameter. +

This allows us to extract one or all fields as idempotency key. + 2. Different from `idempotent` decorator, we must explicitly register the Lambda context to [protect against timeouts](#lambda-timeouts). + === "Using Pydantic" - ```python hl_lines="1-5 10 23 34" + ```python title="working_with_idempotent_function_pydantic.py" hl_lines="3-7 12 26 37" --8<-- "examples/idempotency/src/working_with_idempotent_function_pydantic.py" ``` @@ -166,15 +212,15 @@ By default, `idempotent_function` serializes, stores, and returns your annotated The output serializer supports any JSON serializable data, **Python Dataclasses** and **Pydantic Models**. -!!! info "When using the `output_serializer` parameter, the data will continue to be stored in DynamoDB as a JSON object." +!!! info "When using the `output_serializer` parameter, the data will continue to be stored in your persistent storage as a JSON string." === "Pydantic" - You can use `PydanticSerializer` to automatically serialize what's retrieved from the persistent storage based on the return type annotated. + Use `PydanticSerializer` to automatically serialize what's retrieved from the persistent storage based on the return type annotated. === "Inferring via the return type" - ```python hl_lines="6 24 25 32 36 45" + ```python hl_lines="8 27 35 38 48" --8<-- "examples/idempotency/src/working_with_pydantic_deduced_output_serializer.py" ``` @@ -184,17 +230,17 @@ The output serializer supports any JSON serializable data, **Python Dataclasses* Alternatively, you can provide an explicit model as an input to `PydanticSerializer`. - ```python hl_lines="6 24 25 32 35 44" + ```python hl_lines="8 27 35 35 47" --8<-- "examples/idempotency/src/working_with_pydantic_explicitly_output_serializer.py" ``` === "Dataclasses" - You can use `DataclassSerializer` to automatically serialize what's retrieved from the persistent storage based on the return type annotated. + Use `DataclassSerializer` to automatically serialize what's retrieved from the persistent storage based on the return type annotated. === "Inferring via the return type" - ```python hl_lines="8 27-29 36 40 49" + ```python hl_lines="9 30 38 41 51" --8<-- "examples/idempotency/src/working_with_dataclass_deduced_output_serializer.py" ``` @@ -204,18 +250,18 @@ The output serializer supports any JSON serializable data, **Python Dataclasses* Alternatively, you can provide an explicit model as an input to `DataclassSerializer`. - ```python hl_lines="8 27-29 36 39 48" + ```python hl_lines="8 30 38 40 50" --8<-- "examples/idempotency/src/working_with_dataclass_explicitly_output_serializer.py" ``` === "Any type" - You can use `CustomDictSerializer` to have full control over the serialization process for any type. It expects two functions: + Use `CustomDictSerializer` to have full control over the serialization process for any type. It expects two functions: * **to_dict**. Function to convert any type to a JSON serializable dictionary before it saves into the persistent storage. * **from_dict**. Function to convert from a dictionary retrieved from persistent storage and serialize in its original form. - ```python hl_lines="8 32 36 40 50 53" + ```python hl_lines="9 34 38 42 52 54 64" --8<-- "examples/idempotency/src/working_with_idempotent_function_custom_output_serializer.py" ``` @@ -223,42 +269,42 @@ The output serializer supports any JSON serializable data, **Python Dataclasses* 2. This function does the following

**1**. Receives the dictionary saved into the persistent storage
**1** Serializes to `OrderOutput` before `@idempotent` returns back to the caller. 3. This serializer receives both functions so it knows who to call when to serialize to and from dictionary. -#### Batch integration +### Using in-memory cache -You can can easily integrate with [Batch utility](batch.md){target="_blank"} via context manager. This ensures that you process each record in an idempotent manner, and guard against a [Lambda timeout](#lambda-timeouts) idempotent situation. +!!! note "In-memory cache is local to each Lambda execution environment." -???+ "Choosing an unique batch record attribute" - In this example, we choose `messageId` as our idempotency key since we know it'll be unique. +You can enable caching with the `use_local_cache` parameter in `IdempotencyConfig`. When enabled, you can adjust cache capacity _(256)_ with `local_cache_max_items`. - Depending on your use case, it might be more accurate [to choose another field](#choosing-a-payload-subset-for-idempotency) your producer intentionally set to define uniqueness. +By default, caching is disabled since we don't know how big your response could be in relation to your configured memory size. -=== "Integration with Batch Processor" +=== "Enabling cache" - ```python hl_lines="2 12 16 20 31 35 37" - --8<-- "examples/idempotency/src/integrate_idempotency_with_batch_processor.py" + ```python hl_lines="15" + --8<-- "examples/idempotency/src/working_with_local_cache.py" ``` + 1. You can adjust cache capacity with [`local_cache_max_items`](#customizing-the-default-behavior) parameter. + === "Sample event" - ```json hl_lines="4" - --8<-- "examples/idempotency/src/integrate_idempotency_with_batch_processor_payload.json" + ```json + --8<-- "examples/idempotency/src/working_with_local_cache_payload.json" ``` -### Choosing a payload subset for idempotency +### Choosing a payload subset ???+ tip "Tip: Dealing with always changing payloads" When dealing with a more elaborate payload, where parts of the payload always change, you should use **`event_key_jmespath`** parameter. -Use [`IdempotencyConfig`](#customizing-the-default-behavior) to instruct the idempotent decorator to only use a portion of your payload to verify whether a request is idempotent, and therefore it should not be retried. +Use **`event_key_jmespath`** parameter in [`IdempotencyConfig`](#customizing-the-default-behavior) to select one or more payload parts as your idempotency key. -> **Payment scenario** +> **Example scenario** In this example, we have a Lambda handler that creates a payment for a user subscribing to a product. We want to ensure that we don't accidentally charge our customer by subscribing them more than once. -Imagine the function executes successfully, but the client never receives the response due to a connection issue. It is safe to retry in this instance, as the idempotent decorator will return a previously saved response. +Imagine the function runs successfully, but the client never receives the response due to a connection issue. It is safe to immediately retry in this instance, as the idempotent decorator will return a previously saved response. -**What we want here** is to instruct Idempotency to use `user_id` and `product_id` fields from our incoming payload as our idempotency key. -If we were to treat the entire request as our idempotency key, a simple HTTP header change would cause our customer to be charged twice. +We want to use `user_id` and `product_id` fields as our idempotency key. **If we were** to treat the entire request as our idempotency key, a simple HTTP header change would cause our function to run again. ???+ tip "Deserializing JSON strings in payloads for increased accuracy." The payload extracted by the `event_key_jmespath` is treated as a string by default. @@ -268,7 +314,7 @@ If we were to treat the entire request as our idempotency key, a simple HTTP hea === "Payment function" - ```python hl_lines="5-9 16 30" + ```python hl_lines="6-10 18 31" --8<-- "examples/idempotency/src/working_with_payload_subset.py" ``` @@ -278,68 +324,64 @@ If we were to treat the entire request as our idempotency key, a simple HTTP hea --8<-- "examples/idempotency/src/working_with_payload_subset_payload.json" ``` -### Lambda timeouts +### Adjusting expiration window -???+ note - This is automatically done when you decorate your Lambda handler with [@idempotent decorator](#idempotent-decorator). +!!! note "By default, we expire idempotency records after **an hour** (3600 seconds). After that, a transaction with the same payload [will not be considered idempotent](#expired-idempotency-records)." -To prevent against extended failed retries when a [Lambda function times out](https://aws.amazon.com/premiumsupport/knowledge-center/lambda-verify-invocation-timeouts/){target="_blank"}, -Powertools for AWS Lambda (Python) calculates and includes the remaining invocation available time as part of the idempotency record. +You can change this expiration window with the **`expires_after_seconds`** parameter. There is no limit on how long this expiration window can be set to. -???+ example - If a second invocation happens **after** this timestamp, and the record is marked as `INPROGRESS`, we will execute the invocation again as if it was in the `EXPIRED` state (e.g, `expire_seconds` field elapsed). +=== "Adjusting expiration window" - This means that if an invocation expired during execution, it will be quickly executed again on the next retry. + ```python hl_lines="14" + --8<-- "examples/idempotency/src/working_with_record_expiration.py" + ``` -???+ important - If you are only using the [@idempotent_function decorator](#idempotent_function-decorator) to guard isolated parts of your code, - you must use `register_lambda_context` available in the [idempotency config object](#customizing-the-default-behavior) to benefit from this protection. +=== "Sample event" -Here is an example on how you register the Lambda context in your handler: + ```json + --8<-- "examples/idempotency/src/working_with_record_expiration_payload.json" + ``` -=== "Registering the Lambda context" +???+ important "Idempotency record expiration vs DynamoDB time-to-live (TTL)" + [DynamoDB TTL is a feature](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/howitworks-ttl.html){target="_blank"} to remove items after a certain period of time, it may occur within 48 hours of expiration. - ```python hl_lines="11 20" - --8<-- "examples/idempotency/src/working_with_lambda_timeout.py" - ``` + We don't rely on DynamoDB or any persistence storage layer to determine whether a record is expired to avoid eventual inconsistency states. -### Handling exceptions + Instead, Idempotency records saved in the storage layer contain timestamps that can be verified upon retrieval and double checked within Idempotency feature. -If you are using the `idempotent` decorator on your Lambda handler, any unhandled exceptions that are raised during the code execution will cause **the record in the persistence layer to be deleted**. -This means that new invocations will execute your code again despite having the same payload. If you don't want the record to be deleted, you need to catch exceptions within the idempotent function and return a successful response. + **Why?** -
-```mermaid -sequenceDiagram - participant Client - participant Lambda - participant Persistence Layer - Client->>Lambda: Invoke (event) - Lambda->>Persistence Layer: Get or set (id=event.search(payload)) - activate Persistence Layer - Note right of Persistence Layer: Locked during this time. Prevents multiple
Lambda invocations with the same
payload running concurrently. - Lambda--xLambda: Call handler (event).
Raises exception - Lambda->>Persistence Layer: Delete record (id=event.search(payload)) - deactivate Persistence Layer - Lambda-->>Client: Return error response + A record might still be valid (`COMPLETE`) when we retrieved, but in some rare cases it might expire a second later. A record could also be [cached in memory](#using-in-memory-cache). You might also want to have idempotent transactions that should expire in seconds. + +### Lambda timeouts + +!!! note "You can skip this section if you are using the [`@idempotent` decorator](#idempotent-decorator)" + +By default, we protect against [concurrent executions](#handling-concurrent-executions-with-the-same-payload) with the same payload using a locking mechanism. However, if your Lambda function times out before completing the first invocation it will only accept the same request when the [idempotency record expire](#adjusting-expiration-window). + +To prevent extended failures, use **`register_lambda_context`** function from your idempotency config to calculate and include the remaining invocation time in your idempotency record. + +```python title="working_with_lambda_timeout.py" hl_lines="14 23" +--8<-- "examples/idempotency/src/working_with_lambda_timeout.py" ``` -Idempotent sequence exception -
-If you are using `idempotent_function`, any unhandled exceptions that are raised _inside_ the decorated function will cause the record in the persistence layer to be deleted, and allow the function to be executed again if retried. +???+ example "Mechanics" + If a second invocation happens **after** this timestamp, and the record is marked as `INPROGRESS`, we will run the invocation again as if it was in the `EXPIRED` state. -If an Exception is raised _outside_ the scope of the decorated function and after your function has been called, the persistent record will not be affected. In this case, idempotency will be maintained for your decorated function. Example: + This means that if an invocation expired during execution, it will be quickly executed again on the next retry. -=== "Handling exceptions" +### Handling exceptions - ```python hl_lines="18-22 28 31" - --8<-- "examples/idempotency/src/working_with_exceptions.py" - ``` +There are two failure modes that can cause new invocations to execute your code again despite having the same payload: -???+ warning - **We will raise `IdempotencyPersistenceLayerError`** if any of the calls to the persistence layer fail unexpectedly. +* **Unhandled exception**. We catch them to delete the idempotency record to prevent inconsistencies, then propagate them. +* **Persistent layer errors**. We raise **`IdempotencyPersistenceLayerError`** for any persistence layer errors _e.g., remove idempotency record_. - As this happens outside the scope of your decorated function, you are not able to catch it if you're using the `idempotent` decorator on your Lambda handler. +If an exception is handled or raised **outside** your decorated function, then idempotency will be maintained. + +```python title="working_with_exceptions.py" hl_lines="21 32 38" +--8<-- "examples/idempotency/src/working_with_exceptions.py" +``` ### Persistence layers @@ -347,13 +389,41 @@ If an Exception is raised _outside_ the scope of the decorated function and afte This persistence layer is built-in, allowing you to use an existing DynamoDB table or create a new one dedicated to idempotency state (recommended). -=== "Customizing DynamoDBPersistenceLayer to suit your table structure" +```python title="customize_persistence_layer.py" hl_lines="10-18" +--8<-- "examples/idempotency/src/customize_persistence_layer.py" +``` - ```python hl_lines="7-15" - --8<-- "examples/idempotency/src/customize_persistence_layer.py" +##### Using a composite primary key + +Use `sort_key_attr` parameter when your table is configured with a composite primary key _(hash+range key)_. + +When enabled, we will save the idempotency key in the sort key instead. By default, the primary key will now be set to `idempotency#{LAMBDA_FUNCTION_NAME}`. + +You can optionally set a static value for the partition key using the `static_pk_value` parameter. + +=== "Reusing a DynamoDB table that uses a composite primary key" + + ```python hl_lines="10" + --8<-- "examples/idempotency/src/working_with_composite_key.py" ``` -When using DynamoDB as the persistence layer, you can customize the attribute names by passing the following parameters during the initialization of the persistence layer: +=== "Sample Event" + + ```json + --8<-- "examples/idempotency/src/working_with_composite_key_payload.json" + ``` + +??? note "Click to expand and learn how table items would look like" + + | id | sort_key | expiration | status | data | + | ---------------------------- | -------------------------------- | ---------- | ----------- | ----------------------------------------- | + | idempotency#MyLambdaFunction | 1e956ef7da78d0cb890be999aecc0c9e | 1636549553 | COMPLETED | {"user_id": 12391, "message": "success"} | + | idempotency#MyLambdaFunction | 2b2cdb5f86361e97b4383087c1ffdf27 | 1636549571 | COMPLETED | {"user_id": 527212, "message": "success"} | + | idempotency#MyLambdaFunction | f091d2527ad1c78f05d54cc3f363be80 | 1636549585 | IN_PROGRESS | | + +##### DynamoDB attributes + +You can customize the attribute names during initialization: | Parameter | Required | Default | Description | | --------------------------- | ------------------ | ------------------------------------ | -------------------------------------------------------------------------------------------------------- | @@ -369,22 +439,102 @@ When using DynamoDB as the persistence layer, you can customize the attribute na #### RedisPersistenceLayer -This persistence layer is built-in, allowing you to use an existing Redis service. For optimal performance and compatibility, it is strongly recommended to use a Redis service version 7 or higher. +!!! info "We recommend Redis version 7 or higher for optimal performance." + +For simple setups, initialize `RedisCachePersistenceLayer` with your Redis endpoint and port to connect. + +For security, we enforce SSL connections by default; to disable it, set `ssl=False`. -=== "Customizing RedisPersistenceLayer to suit your data structure" +=== "Redis quick start" + ```python title="getting_started_with_idempotency_redis_config.py" hl_lines="8-10 14 27" + --8<-- "examples/idempotency/src/getting_started_with_idempotency_redis_config.py" + ``` - ```python hl_lines="9-16" - --8<-- "examples/idempotency/src/customize_persistence_layer_redis.py" +=== "Using an existing Redis client" + ```python title="getting_started_with_idempotency_redis_client.py" hl_lines="5 10-11 16 24 38" + --8<-- "examples/idempotency/src/getting_started_with_idempotency_redis_client.py" ``` -When using Redis as the persistence layer, you can customize the attribute names by providing the following parameters upon initialization of the persistence layer: +=== "Sample event" -| Parameter | Required | Default | Description | -| --------------------------- | ------------------ | ------------------------------------ | -------------------------------------------------------------------------------------------------------- | -| **in_progress_expiry_attr** | | `in_progress_expiration` | Unix timestamp of when record expires while in progress (in case of the invocation times out) | -| **status_attr** | | `status` | Stores status of the Lambda execution during and after invocation | -| **data_attr** | | `data` | Stores results of successfully executed Lambda handlers | -| **validation_key_attr** | | `validation` | Hashed representation of the parts of the event used for validation | + ```json title="getting_started_with_idempotency_payload.json" + --8<-- "examples/idempotency/src/getting_started_with_idempotency_payload.json" + ``` + +##### Redis SSL connections + +We recommend using AWS Secrets Manager to store and rotate certificates safely, and the [Parameters feature](./parameters.md){target="_blank"} to fetch and cache optimally. + +For advanced configurations, we recommend using an existing Redis client for optimal compatibility like SSL certificates and timeout. + +=== "Advanced configuration using AWS Secrets" + ```python title="using_redis_client_with_aws_secrets.py" hl_lines="9-11 13 15 25" + --8<-- "examples/idempotency/src/using_redis_client_with_aws_secrets.py" + ``` + + 1. JSON stored: + ```json + { + "REDIS_ENDPOINT": "127.0.0.1", + "REDIS_PORT": "6379", + "REDIS_PASSWORD": "redis-secret" + } + ``` + +=== "Advanced configuration with local certificates" + ```python title="using_redis_client_with_local_certs.py" hl_lines="14 25-27" + --8<-- "examples/idempotency/src/using_redis_client_with_local_certs.py" + ``` + + 1. JSON stored: + ```json + { + "REDIS_ENDPOINT": "127.0.0.1", + "REDIS_PORT": "6379", + "REDIS_PASSWORD": "redis-secret" + } + ``` + 2. redis_user.crt file stored in the "certs" directory of your Lambda function + 3. redis_user_private.key file stored in the "certs" directory of your Lambda function + 4. redis_ca.pem file stored in the "certs" directory of your Lambda function + +##### Redis attributes + +You can customize the attribute names during initialization: + +| Parameter | Required | Default | Description | +| --------------------------- | -------- | ------------------------ | --------------------------------------------------------------------------------------------- | +| **in_progress_expiry_attr** | | `in_progress_expiration` | Unix timestamp of when record expires while in progress (in case of the invocation times out) | +| **status_attr** | | `status` | Stores status of the Lambda execution during and after invocation | +| **data_attr** | | `data` | Stores results of successfully executed Lambda handlers | +| **validation_key_attr** | | `validation` | Hashed representation of the parts of the event used for validation | + +```python title="customize_persistence_layer_redis.py" hl_lines="15-18" +--8<-- "examples/idempotency/src/customize_persistence_layer_redis.py" +``` + +### Common use cases + +#### Batch processing + +You can can easily integrate with [Batch](batch.md){target="_blank"} using the [idempotent_function decorator](#idempotent_function-decorator) to handle idempotency per message/record in a given batch. + +???+ "Choosing an unique batch record attribute" + In this example, we choose `messageId` as our idempotency key since we know it'll be unique. + + Depending on your use case, it might be more accurate [to choose another field](#choosing-a-payload-subset) your producer intentionally set to define uniqueness. + +=== "Integration with Batch Processor" + + ```python title="integrate_idempotency_with_batch_processor.py" hl_lines="3 16 19 25 27" + --8<-- "examples/idempotency/src/integrate_idempotency_with_batch_processor.py" + ``` + +=== "Sample event" + + ```json title="integrate_idempotency_with_batch_processor_payload.json" hl_lines="4" + --8<-- "examples/idempotency/src/integrate_idempotency_with_batch_processor_payload.json" + ``` ### Idempotency request flow @@ -551,6 +701,26 @@ sequenceDiagram Concurrent identical in-flight requests
+#### Unhandled exception + +
+```mermaid +sequenceDiagram + participant Client + participant Lambda + participant Persistence Layer + Client->>Lambda: Invoke (event) + Lambda->>Persistence Layer: Get or set (id=event.search(payload)) + activate Persistence Layer + Note right of Persistence Layer: Locked during this time. Prevents multiple
Lambda invocations with the same
payload running concurrently. + Lambda--xLambda: Call handler (event).
Raises exception + Lambda->>Persistence Layer: Delete record (id=event.search(payload)) + deactivate Persistence Layer + Lambda-->>Client: Return error response +``` +Idempotent sequence exception +
+ #### Lambda request timeout
@@ -638,110 +808,21 @@ graph TD; Race condition with Redis
-## Redis as persistent storage layer provider - -### Redis resources - -Before setting up Redis as the persistent storage layer provider, you must have an existing Redis service. We recommend you to use Redis compatible services such as [Amazon ElastiCache for Redis](https://aws.amazon.com/elasticache/redis/){target="_blank"} or [Amazon MemoryDB for Redis](https://aws.amazon.com/memorydb/){target="_blank"} as your persistent storage layer provider. - -???+ tip "No existing Redis service?" - If you don't have an existing Redis service, we recommend using [DynamoDB](#dynamodbpersistencelayer) as the persistent storage layer provider. - -=== "AWS CloudFormation example" - - ```yaml hl_lines="5" - --8<-- "examples/idempotency/templates/cfn_redis_serverless.yaml" - ``` - - 1. Replace the Security Group ID and Subnet ID to match your VPC settings. - -### VPC Access - -Your Lambda Function must have network access to the Redis endpoint before using it as the idempotency persistent storage layer. In most cases, you will need to [configure VPC access](https://docs.aws.amazon.com/lambda/latest/dg/configuration-vpc.html){target="_blank"} for your Lambda Function. - -???+ tip "Amazon ElastiCache/MemoryDB for Redis as persistent storage layer provider" - If you plan to use Amazon ElastiCache for Redis as the idempotency persistent storage layer, you may find [this AWS tutorial](https://docs.aws.amazon.com/lambda/latest/dg/services-elasticache-tutorial.html){target="_blank"} helpful. - For those using Amazon MemoryDB for Redis, refer to [this AWS tutorial](https://aws.amazon.com/blogs/database/access-amazon-memorydb-for-redis-from-aws-lambda/){target="_blank"} specifically for the VPC setup guidance. - -After completing the VPC setup, you can use the templates provided below to set up Lambda functions with access to VPC internal subnets. - -=== "AWS Serverless Application Model (SAM) example" - - ```yaml hl_lines="9" - --8<-- "examples/idempotency/templates/sam_redis_vpc.yaml" - ``` - - 1. Replace the Security Group ID and Subnet ID to match your VPC settings. - -### Configuring Redis persistence layer - -You can quickly get started by initializing the `RedisCachePersistenceLayer` class and applying the `idempotent` decorator to your Lambda handler. For a detailed example of using the `RedisCachePersistenceLayer`, refer to the [Persistence layers section](#redispersistencelayer). - -???+ info - We enforce security best practices by using SSL connections in the `RedisCachePersistenceLayer`; to disable it, set `ssl=False` - -=== "Use Persistence Layer with Redis config variables" - ```python hl_lines="7-9 12 26" - --8<-- "examples/idempotency/src/getting_started_with_idempotency_redis_config.py" - ``` - -=== "Use established Redis Client" - ```python hl_lines="4 9-11 14 22 36" - --8<-- "examples/idempotency/src/getting_started_with_idempotency_redis_client.py" - ``` - -=== "Sample event" - - ```json - --8<-- "examples/idempotency/src/getting_started_with_idempotency_payload.json" - ``` - -### Custom advanced settings - -For advanced configurations, such as setting up SSL certificates or customizing parameters like a custom timeout, you can utilize the Redis client to tailor these specific settings to your needs. - -=== "Advanced configuration using AWS Secrets" - ```python hl_lines="7-9 11 13 23" - --8<-- "examples/idempotency/src/using_redis_client_with_aws_secrets.py" - ``` - - 1. JSON stored: - { - "REDIS_ENDPOINT": "127.0.0.1", - "REDIS_PORT": "6379", - "REDIS_PASSWORD": "redis-secret" - } - -=== "Advanced configuration with local certificates" - ```python hl_lines="12 23-25" - --8<-- "examples/idempotency/src/using_redis_client_with_local_certs.py" - ``` - - 1. JSON stored: - { - "REDIS_ENDPOINT": "127.0.0.1", - "REDIS_PORT": "6379", - "REDIS_PASSWORD": "redis-secret" - } - 2. redis_user.crt file stored in the "certs" directory of your Lambda function - 3. redis_user_private.key file stored in the "certs" directory of your Lambda function - 4. redis_ca.pem file stored in the "certs" directory of your Lambda function - ## Advanced ### Customizing the default behavior -Idempotent decorator can be further configured with **`IdempotencyConfig`** as seen in the previous example. These are the available options for further configuration - -| Parameter | Default | Description | -|---------------------------------|---------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| **event_key_jmespath** | `""` | JMESPath expression to extract the idempotency key from the event record using [built-in functions](./jmespath_functions.md#built-in-jmespath-functions){target="_blank"} | -| **payload_validation_jmespath** | `""` | JMESPath expression to validate whether certain parameters have changed in the event while the event payload | -| **raise_on_no_idempotency_key** | `False` | Raise exception if no idempotency key was found in the request | -| **expires_after_seconds** | 3600 | The number of seconds to wait before a record is expired | -| **use_local_cache** | `False` | Whether to locally cache idempotency results | -| **local_cache_max_items** | 256 | Max number of items to store in local cache | -| **hash_function** | `md5` | Function to use for calculating hashes, as provided by [hashlib](https://docs.python.org/3/library/hashlib.html){target="_blank" rel="nofollow"} in the standard library. | +You can override and further extend idempotency behavior via **`IdempotencyConfig`** with the following options: + +| Parameter | Default | Description | +| ------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| **event_key_jmespath** | `""` | JMESPath expression to extract the idempotency key from the event record using [built-in functions](./jmespath_functions.md#built-in-jmespath-functions){target="_blank"} | +| **payload_validation_jmespath** | `""` | JMESPath expression to validate whether certain parameters have changed in the event while the event payload _e.g., payload tampering._ | +| **raise_on_no_idempotency_key** | `False` | Raise exception if no idempotency key was found in the request | +| **expires_after_seconds** | 3600 | The number of seconds to wait before a record is expired, allowing a new transaction with the same idempotency key | +| **use_local_cache** | `False` | Whether to cache idempotency results in-memory to save on persistence storage latency and costs | +| **local_cache_max_items** | 256 | Max number of items to store in local cache | +| **hash_function** | `md5` | Function to use for calculating hashes, as provided by [hashlib](https://docs.python.org/3/library/hashlib.html){target="_blank" rel="nofollow"} in the standard library. | | **response_hook** | `None` | Function to use for processing the stored Idempotent response. This function hook is called when an existing idempotent response is found. See [Manipulating The Idempotent Response](idempotency.md#manipulating-the-idempotent-response) | ### Handling concurrent executions with the same payload @@ -753,62 +834,6 @@ This utility will raise an **`IdempotencyAlreadyInProgressError`** exception if This is a locking mechanism for correctness. Since we don't know the result from the first invocation yet, we can't safely allow another concurrent execution. -### Using in-memory cache - -**By default, in-memory local caching is disabled**, since we don't know how much memory you consume per invocation compared to the maximum configured in your Lambda function. - -???+ note "Note: This in-memory cache is local to each Lambda execution environment" - This means it will be effective in cases where your function's concurrency is low in comparison to the number of "retry" invocations with the same payload, because cache might be empty. - -You can enable in-memory caching with the **`use_local_cache`** parameter: - -=== "Caching idempotent transactions in-memory to prevent multiple calls to storage" - - ```python hl_lines="11" - --8<-- "examples/idempotency/src/working_with_local_cache.py" - ``` - -=== "Sample event" - - ```json - --8<-- "examples/idempotency/src/working_with_local_cache_payload.json" - ``` - -When enabled, the default is to cache a maximum of 256 records in each Lambda execution environment - You can change it with the **`local_cache_max_items`** parameter. - -### Expiring idempotency records - -!!! note "By default, we expire idempotency records after **an hour** (3600 seconds)." - -In most cases, it is not desirable to store the idempotency records forever. Rather, you want to guarantee that the same payload won't be executed within a period of time. - -You can change this window with the **`expires_after_seconds`** parameter: - -=== "Adjusting idempotency record expiration" - - ```python hl_lines="11" - --8<-- "examples/idempotency/src/working_with_record_expiration.py" - ``` - -=== "Sample event" - - ```json - --8<-- "examples/idempotency/src/working_with_record_expiration_payload.json" - ``` - -This will mark any records older than 5 minutes as expired, and [your function will be executed as normal if it is invoked with a matching payload](#expired-idempotency-records). - -???+ important "Idempotency record expiration vs DynamoDB time-to-live (TTL)" - [DynamoDB TTL is a feature](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/howitworks-ttl.html){target="_blank"} to remove items after a certain period of time, it may occur within 48 hours of expiration. - - We don't rely on DynamoDB or any persistence storage layer to determine whether a record is expired to avoid eventual inconsistency states. - - Instead, Idempotency records saved in the storage layer contain timestamps that can be verified upon retrieval and double checked within Idempotency feature. - - **Why?** - - A record might still be valid (`COMPLETE`) when we retrieved, but in some rare cases it might expire a second later. A record could also be [cached in memory](#using-in-memory-cache). You might also want to have idempotent transactions that should expire in seconds. - ### Payload validation ???+ question "Question: What if your function is invoked with the same payload except some outer parameters have changed?" @@ -820,7 +845,7 @@ With **`payload_validation_jmespath`**, you can provide an additional JMESPath e === "Payload validation" - ```python hl_lines="12 20 28" + ```python hl_lines="20 29 36" --8<-- "examples/idempotency/src/working_with_validation_payload.py" ``` @@ -856,7 +881,7 @@ This means that we will raise **`IdempotencyKeyError`** if the evaluation of **` === "Idempotency key required" - ```python hl_lines="11" + ```python hl_lines="14" --8<-- "examples/idempotency/src/working_with_idempotency_key_required.py" ``` @@ -878,13 +903,13 @@ The **`boto_config`** and **`boto3_session`** parameters enable you to pass in a === "Custom session" - ```python hl_lines="1 11 13" + ```python hl_lines="3 13 16" --8<-- "examples/idempotency/src/working_with_custom_session.py" ``` === "Custom config" - ```python hl_lines="1 11 13" + ```python hl_lines="3 13 16" --8<-- "examples/idempotency/src/working_with_custom_config.py" ``` @@ -894,34 +919,6 @@ The **`boto_config`** and **`boto3_session`** parameters enable you to pass in a --8<-- "examples/idempotency/src/working_with_custom_config_payload.json" ``` -### Using a DynamoDB table with a composite primary key - -When using a composite primary key table (hash+range key), use `sort_key_attr` parameter when initializing your persistence layer. - -With this setting, we will save the idempotency key in the sort key instead of the primary key. By default, the primary key will now be set to `idempotency#{LAMBDA_FUNCTION_NAME}`. - -You can optionally set a static value for the partition key using the `static_pk_value` parameter. - -=== "Reusing a DynamoDB table that uses a composite primary key" - - ```python hl_lines="7" - --8<-- "examples/idempotency/src/working_with_composite_key.py" - ``` - -=== "Sample Event" - - ```json - --8<-- "examples/idempotency/src/working_with_composite_key_payload.json" - ``` - -The example function above would cause data to be stored in DynamoDB like this: - -| id | sort_key | expiration | status | data | -| ---------------------------- | -------------------------------- | ---------- | ----------- | ----------------------------------------- | -| idempotency#MyLambdaFunction | 1e956ef7da78d0cb890be999aecc0c9e | 1636549553 | COMPLETED | {"user_id": 12391, "message": "success"} | -| idempotency#MyLambdaFunction | 2b2cdb5f86361e97b4383087c1ffdf27 | 1636549571 | COMPLETED | {"user_id": 527212, "message": "success"} | -| idempotency#MyLambdaFunction | f091d2527ad1c78f05d54cc3f363be80 | 1636549585 | IN_PROGRESS | | - ### Bring your own persistent store This utility provides an abstract base class (ABC), so that you can implement your choice of persistent storage layer. @@ -933,11 +930,9 @@ You can create your own persistent store from scratch by inheriting the `BasePer * **`_update_record()`** – Updates an item in the persistence store. * **`_delete_record()`** – Removes an item from the persistence store. -=== "Bring your own persistent store" - - ```python hl_lines="8 18 65 74 96 124" - --8<-- "examples/idempotency/src/bring_your_own_persistent_store.py" - ``` +```python title="bring_your_own_persistent_store.py" hl_lines="8 18 65 74 96 124" +--8<-- "examples/idempotency/src/bring_your_own_persistent_store.py" +``` ???+ danger Pay attention to the documentation for each - you may need to perform additional checks inside these methods to ensure the idempotency guarantees remain intact. @@ -950,7 +945,7 @@ You can set up a `response_hook` in the `IdempotentConfig` class to manipulate t === "Using an Idempotent Response Hook" - ```python hl_lines="19 21 27 34" + ```python hl_lines="20 22 28 36" --8<-- "examples/idempotency/src/working_with_response_hook.py" ``` @@ -976,11 +971,7 @@ When using response hooks to manipulate returned data from idempotent operations ## Compatibility with other utilities -### Batch - -See [Batch integration](#batch-integration) above. - -### Validation utility +### JSON Schema Validation The idempotency utility can be used with the `validator` decorator. Ensure that idempotency is the innermost decorator. @@ -990,9 +981,9 @@ The idempotency utility can be used with the `validator` decorator. Ensure that Make sure to account for this behavior, if you set the `event_key_jmespath`. -=== "Using Idempotency with JSONSchema Validation utility" +=== "Using Idempotency with validation utility" - ```python hl_lines="13" + ```python hl_lines="16" --8<-- "examples/idempotency/src/integrate_idempotency_with_validator.py" ``` diff --git a/examples/idempotency/src/customize_persistence_layer.py b/examples/idempotency/src/customize_persistence_layer.py index 26409191ca9..a4e9aa6993e 100644 --- a/examples/idempotency/src/customize_persistence_layer.py +++ b/examples/idempotency/src/customize_persistence_layer.py @@ -1,11 +1,14 @@ +import os + from aws_lambda_powertools.utilities.idempotency import ( DynamoDBPersistenceLayer, idempotent, ) from aws_lambda_powertools.utilities.typing import LambdaContext +table = os.getenv("IDEMPOTENCY_TABLE", "") persistence_layer = DynamoDBPersistenceLayer( - table_name="IdempotencyTable", + table_name=table, key_attr="idempotency_key", expiry_attr="expires_at", in_progress_expiry_attr="in_progress_expires_at", diff --git a/examples/idempotency/src/customize_persistence_layer_redis.py b/examples/idempotency/src/customize_persistence_layer_redis.py index 7db3d1b53ea..40aef433396 100644 --- a/examples/idempotency/src/customize_persistence_layer_redis.py +++ b/examples/idempotency/src/customize_persistence_layer_redis.py @@ -1,3 +1,5 @@ +import os + from aws_lambda_powertools.utilities.idempotency import ( idempotent, ) @@ -6,8 +8,9 @@ ) from aws_lambda_powertools.utilities.typing import LambdaContext +redis_endpoint = os.getenv("REDIS_CLUSTER_ENDPOINT", "localhost") persistence_layer = RedisCachePersistenceLayer( - host="localhost", + host=redis_endpoint, port=6379, in_progress_expiry_attr="in_progress_expiration", status_attr="status", diff --git a/examples/idempotency/src/getting_started_with_idempotency.py b/examples/idempotency/src/getting_started_with_idempotency.py index 0754f42c6b3..b17426c06f2 100644 --- a/examples/idempotency/src/getting_started_with_idempotency.py +++ b/examples/idempotency/src/getting_started_with_idempotency.py @@ -1,3 +1,4 @@ +import os from dataclasses import dataclass, field from uuid import uuid4 @@ -7,7 +8,8 @@ ) from aws_lambda_powertools.utilities.typing import LambdaContext -persistence_layer = DynamoDBPersistenceLayer(table_name="IdempotencyTable") +table = os.getenv("IDEMPOTENCY_TABLE", "") +persistence_layer = DynamoDBPersistenceLayer(table_name=table) @dataclass @@ -17,8 +19,7 @@ class Payment: payment_id: str = field(default_factory=lambda: f"{uuid4()}") -class PaymentError(Exception): - ... +class PaymentError(Exception): ... @idempotent(persistence_store=persistence_layer) diff --git a/examples/idempotency/src/getting_started_with_idempotency_redis_client.py b/examples/idempotency/src/getting_started_with_idempotency_redis_client.py index f06d059fad4..24dfe1be117 100644 --- a/examples/idempotency/src/getting_started_with_idempotency_redis_client.py +++ b/examples/idempotency/src/getting_started_with_idempotency_redis_client.py @@ -1,3 +1,4 @@ +import os from dataclasses import dataclass, field from uuid import uuid4 @@ -11,8 +12,9 @@ ) from aws_lambda_powertools.utilities.typing import LambdaContext +redis_endpoint = os.getenv("REDIS_CLUSTER_ENDPOINT", "localhost") client = Redis( - host="localhost", + host=redis_endpoint, port=6379, socket_connect_timeout=5, socket_timeout=5, @@ -29,8 +31,7 @@ class Payment: payment_id: str = field(default_factory=lambda: f"{uuid4()}") -class PaymentError(Exception): - ... +class PaymentError(Exception): ... @idempotent(persistence_store=persistence_layer) diff --git a/examples/idempotency/src/getting_started_with_idempotency_redis_config.py b/examples/idempotency/src/getting_started_with_idempotency_redis_config.py index de9c6526059..f3917042b28 100644 --- a/examples/idempotency/src/getting_started_with_idempotency_redis_config.py +++ b/examples/idempotency/src/getting_started_with_idempotency_redis_config.py @@ -1,3 +1,4 @@ +import os from dataclasses import dataclass, field from uuid import uuid4 @@ -9,7 +10,8 @@ ) from aws_lambda_powertools.utilities.typing import LambdaContext -persistence_layer = RedisCachePersistenceLayer(host="localhost", port=6379) +redis_endpoint = os.getenv("REDIS_CLUSTER_ENDPOINT", "localhost") +persistence_layer = RedisCachePersistenceLayer(host=redis_endpoint, port=6379) @dataclass @@ -19,8 +21,7 @@ class Payment: payment_id: str = field(default_factory=lambda: f"{uuid4()}") -class PaymentError(Exception): - ... +class PaymentError(Exception): ... @idempotent(persistence_store=persistence_layer) diff --git a/examples/idempotency/src/integrate_idempotency_with_batch_processor.py b/examples/idempotency/src/integrate_idempotency_with_batch_processor.py index 957cefb3202..120c8f12da9 100644 --- a/examples/idempotency/src/integrate_idempotency_with_batch_processor.py +++ b/examples/idempotency/src/integrate_idempotency_with_batch_processor.py @@ -1,5 +1,7 @@ -from aws_lambda_powertools import Logger -from aws_lambda_powertools.utilities.batch import BatchProcessor, EventType +import os +from typing import Any, Dict + +from aws_lambda_powertools.utilities.batch import BatchProcessor, EventType, process_partial_response from aws_lambda_powertools.utilities.data_classes.sqs_event import SQSRecord from aws_lambda_powertools.utilities.idempotency import ( DynamoDBPersistenceLayer, @@ -8,13 +10,11 @@ ) from aws_lambda_powertools.utilities.typing import LambdaContext -logger = Logger() processor = BatchProcessor(event_type=EventType.SQS) -dynamodb = DynamoDBPersistenceLayer(table_name="IdempotencyTable") -config = IdempotencyConfig( - event_key_jmespath="messageId", # see Choosing a payload subset section -) +table = os.getenv("IDEMPOTENCY_TABLE", "") +dynamodb = DynamoDBPersistenceLayer(table_name=table) +config = IdempotencyConfig(event_key_jmespath="messageId") @idempotent_function(data_keyword_argument="record", config=config, persistence_store=dynamodb) @@ -22,16 +22,12 @@ def record_handler(record: SQSRecord): return {"message": record.body} -def lambda_handler(event: SQSRecord, context: LambdaContext): +def lambda_handler(event: Dict[str, Any], context: LambdaContext): config.register_lambda_context(context) # see Lambda timeouts section - # with Lambda context registered for Idempotency - # we can now kick in the Bach processing logic - batch = event["Records"] - with processor(records=batch, handler=record_handler): - # in case you want to access each record processed by your record_handler - # otherwise ignore the result variable assignment - processed_messages = processor.process() - logger.info(processed_messages) - - return processor.response() + return process_partial_response( + event=event, + context=context, + processor=processor, + record_handler=record_handler, + ) diff --git a/examples/idempotency/src/integrate_idempotency_with_validator.py b/examples/idempotency/src/integrate_idempotency_with_validator.py index af833951446..675dbd249a9 100644 --- a/examples/idempotency/src/integrate_idempotency_with_validator.py +++ b/examples/idempotency/src/integrate_idempotency_with_validator.py @@ -1,3 +1,5 @@ +import os + from aws_lambda_powertools.utilities.idempotency import ( DynamoDBPersistenceLayer, IdempotencyConfig, @@ -6,8 +8,9 @@ from aws_lambda_powertools.utilities.typing import LambdaContext from aws_lambda_powertools.utilities.validation import envelopes, validator +table = os.getenv("IDEMPOTENCY_TABLE", "") config = IdempotencyConfig(event_key_jmespath='["message", "username"]') -persistence_layer = DynamoDBPersistenceLayer(table_name="IdempotencyTable") +persistence_layer = DynamoDBPersistenceLayer(table_name=table) @validator(envelope=envelopes.API_GATEWAY_HTTP) diff --git a/examples/idempotency/src/using_redis_client_with_aws_secrets.py b/examples/idempotency/src/using_redis_client_with_aws_secrets.py index f30751c8808..ee9e6d78c45 100644 --- a/examples/idempotency/src/using_redis_client_with_aws_secrets.py +++ b/examples/idempotency/src/using_redis_client_with_aws_secrets.py @@ -1,3 +1,5 @@ +from __future__ import annotations + from typing import Any from redis import Redis @@ -8,11 +10,11 @@ RedisCachePersistenceLayer, ) -redis_values: Any = parameters.get_secret("redis_info", transform="json") # (1)! +redis_values: dict[str, Any] = parameters.get_secret("redis_info", transform="json") # (1)! redis_client = Redis( - host=redis_values.get("REDIS_HOST"), - port=redis_values.get("REDIS_PORT"), + host=redis_values.get("REDIS_HOST", "localhost"), + port=redis_values.get("REDIS_PORT", 6379), password=redis_values.get("REDIS_PASSWORD"), decode_responses=True, socket_timeout=10.0, diff --git a/examples/idempotency/src/using_redis_client_with_local_certs.py b/examples/idempotency/src/using_redis_client_with_local_certs.py index cbad1cc92f4..2b6a5892c5b 100644 --- a/examples/idempotency/src/using_redis_client_with_local_certs.py +++ b/examples/idempotency/src/using_redis_client_with_local_certs.py @@ -1,3 +1,5 @@ +from __future__ import annotations + from typing import Any from redis import Redis @@ -9,12 +11,12 @@ RedisCachePersistenceLayer, ) -redis_values: Any = parameters.get_secret("redis_info", transform="json") # (1)! +redis_values: dict[str, Any] = parameters.get_secret("redis_info", transform="json") # (1)! redis_client = Redis( - host=redis_values.get("REDIS_HOST"), - port=redis_values.get("REDIS_PORT"), + host=redis_values.get("REDIS_HOST", "localhost"), + port=redis_values.get("REDIS_PORT", 6379), password=redis_values.get("REDIS_PASSWORD"), decode_responses=True, socket_timeout=10.0, diff --git a/examples/idempotency/src/working_with_composite_key.py b/examples/idempotency/src/working_with_composite_key.py index f1b70cba99a..92bf1e6ec9a 100644 --- a/examples/idempotency/src/working_with_composite_key.py +++ b/examples/idempotency/src/working_with_composite_key.py @@ -1,10 +1,13 @@ +import os + from aws_lambda_powertools.utilities.idempotency import ( DynamoDBPersistenceLayer, idempotent, ) from aws_lambda_powertools.utilities.typing import LambdaContext -persistence_layer = DynamoDBPersistenceLayer(table_name="IdempotencyTable", sort_key_attr="sort_key") +table = os.getenv("IDEMPOTENCY_TABLE", "") +persistence_layer = DynamoDBPersistenceLayer(table_name=table, sort_key_attr="sort_key") @idempotent(persistence_store=persistence_layer) diff --git a/examples/idempotency/src/working_with_custom_config.py b/examples/idempotency/src/working_with_custom_config.py index 30539f88f3c..3d0f464a1dd 100644 --- a/examples/idempotency/src/working_with_custom_config.py +++ b/examples/idempotency/src/working_with_custom_config.py @@ -1,3 +1,5 @@ +import os + from botocore.config import Config from aws_lambda_powertools.utilities.idempotency import ( @@ -10,7 +12,8 @@ # See: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html#botocore-config boto_config = Config() -persistence_layer = DynamoDBPersistenceLayer(table_name="IdempotencyTable", boto_config=boto_config) +table = os.getenv("IDEMPOTENCY_TABLE", "") +persistence_layer = DynamoDBPersistenceLayer(table_name=table, boto_config=boto_config) config = IdempotencyConfig(event_key_jmespath="body") diff --git a/examples/idempotency/src/working_with_custom_session.py b/examples/idempotency/src/working_with_custom_session.py index aae89f8a3fe..af414c829de 100644 --- a/examples/idempotency/src/working_with_custom_session.py +++ b/examples/idempotency/src/working_with_custom_session.py @@ -1,3 +1,5 @@ +import os + import boto3 from aws_lambda_powertools.utilities.idempotency import ( @@ -10,7 +12,8 @@ # See: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html#module-boto3.session boto3_session = boto3.session.Session() -persistence_layer = DynamoDBPersistenceLayer(table_name="IdempotencyTable", boto3_session=boto3_session) +table = os.getenv("IDEMPOTENCY_TABLE", "") +persistence_layer = DynamoDBPersistenceLayer(table_name=table, boto3_session=boto3_session) config = IdempotencyConfig(event_key_jmespath="body") diff --git a/examples/idempotency/src/working_with_dataclass_deduced_output_serializer.py b/examples/idempotency/src/working_with_dataclass_deduced_output_serializer.py index c59c8b078f7..e6f74cb8f9a 100644 --- a/examples/idempotency/src/working_with_dataclass_deduced_output_serializer.py +++ b/examples/idempotency/src/working_with_dataclass_deduced_output_serializer.py @@ -1,3 +1,4 @@ +import os from dataclasses import dataclass from aws_lambda_powertools.utilities.idempotency import ( @@ -8,7 +9,8 @@ from aws_lambda_powertools.utilities.idempotency.serialization.dataclass import DataclassSerializer from aws_lambda_powertools.utilities.typing import LambdaContext -dynamodb = DynamoDBPersistenceLayer(table_name="IdempotencyTable") +table = os.getenv("IDEMPOTENCY_TABLE", "") +dynamodb = DynamoDBPersistenceLayer(table_name=table) config = IdempotencyConfig(event_key_jmespath="order_id") # see Choosing a payload subset section diff --git a/examples/idempotency/src/working_with_dataclass_explicitly_output_serializer.py b/examples/idempotency/src/working_with_dataclass_explicitly_output_serializer.py index fc2412fb1a2..05ea956d696 100644 --- a/examples/idempotency/src/working_with_dataclass_explicitly_output_serializer.py +++ b/examples/idempotency/src/working_with_dataclass_explicitly_output_serializer.py @@ -1,3 +1,4 @@ +import os from dataclasses import dataclass from aws_lambda_powertools.utilities.idempotency import ( @@ -8,7 +9,8 @@ from aws_lambda_powertools.utilities.idempotency.serialization.dataclass import DataclassSerializer from aws_lambda_powertools.utilities.typing import LambdaContext -dynamodb = DynamoDBPersistenceLayer(table_name="IdempotencyTable") +table = os.getenv("IDEMPOTENCY_TABLE", "") +dynamodb = DynamoDBPersistenceLayer(table_name=table) config = IdempotencyConfig(event_key_jmespath="order_id") # see Choosing a payload subset section diff --git a/examples/idempotency/src/working_with_exceptions.py b/examples/idempotency/src/working_with_exceptions.py index ff282d5a601..b416a61b60a 100644 --- a/examples/idempotency/src/working_with_exceptions.py +++ b/examples/idempotency/src/working_with_exceptions.py @@ -1,3 +1,5 @@ +import os + import requests from aws_lambda_powertools.utilities.idempotency import ( @@ -5,33 +7,32 @@ IdempotencyConfig, idempotent_function, ) +from aws_lambda_powertools.utilities.idempotency.exceptions import IdempotencyPersistenceLayerError from aws_lambda_powertools.utilities.typing import LambdaContext -persistence_layer = DynamoDBPersistenceLayer(table_name="IdempotencyTable") +table = os.getenv("IDEMPOTENCY_TABLE", "") +persistence_layer = DynamoDBPersistenceLayer(table_name=table) config = IdempotencyConfig() -def lambda_handler(event: dict, context: LambdaContext): - # If an exception is raised here, no idempotent record will ever get created as the - # idempotent function does not get called - try: - endpoint = "https://jsonplaceholder.typicode.com/comments/" # change this endpoint to force an exception - requests.get(endpoint) - except Exception as exc: - return str(exc) - - call_external_service(data={"user": "user1", "id": 5}) - - # This exception will not cause the idempotent record to be deleted, since it - # happens after the decorated function has been successfully called - raise Exception - - @idempotent_function(data_keyword_argument="data", config=config, persistence_store=persistence_layer) def call_external_service(data: dict): + # Any exception raised will lead to idempotency record to be deleted result: requests.Response = requests.post( "https://jsonplaceholder.typicode.com/comments/", - json={"user": data["user"], "transaction_id": data["id"]}, + json=data, ) return result.json() + + +def lambda_handler(event: dict, context: LambdaContext): + try: + call_external_service(data=event) + except IdempotencyPersistenceLayerError as e: + # No idempotency, but you can decide to error differently. + raise RuntimeError(f"Oops, can't talk to persistence layer. Permissions? error: {e}") + + # This exception will not impact the idempotency of 'call_external_service' + # because it happens in isolation, or outside their scope. + raise SyntaxError("Oops, this shouldn't be here.") diff --git a/examples/idempotency/src/working_with_idempotency_key_required.py b/examples/idempotency/src/working_with_idempotency_key_required.py index 347740ab4a3..465a7d47e0a 100644 --- a/examples/idempotency/src/working_with_idempotency_key_required.py +++ b/examples/idempotency/src/working_with_idempotency_key_required.py @@ -1,3 +1,5 @@ +import os + from aws_lambda_powertools.utilities.idempotency import ( DynamoDBPersistenceLayer, IdempotencyConfig, @@ -5,7 +7,8 @@ ) from aws_lambda_powertools.utilities.typing import LambdaContext -persistence_layer = DynamoDBPersistenceLayer(table_name="IdempotencyTable") +table = os.getenv("IDEMPOTENCY_TABLE", "") +persistence_layer = DynamoDBPersistenceLayer(table_name=table) config = IdempotencyConfig( event_key_jmespath='["user.uid", "order_id"]', raise_on_no_idempotency_key=True, diff --git a/examples/idempotency/src/working_with_idempotent_function_custom_output_serializer.py b/examples/idempotency/src/working_with_idempotent_function_custom_output_serializer.py index a62961fa5f3..5d6c1ea3b99 100644 --- a/examples/idempotency/src/working_with_idempotent_function_custom_output_serializer.py +++ b/examples/idempotency/src/working_with_idempotent_function_custom_output_serializer.py @@ -1,3 +1,4 @@ +import os from typing import Dict, Type from aws_lambda_powertools.utilities.idempotency import ( @@ -8,7 +9,8 @@ from aws_lambda_powertools.utilities.idempotency.serialization.custom_dict import CustomDictSerializer from aws_lambda_powertools.utilities.typing import LambdaContext -dynamodb = DynamoDBPersistenceLayer(table_name="IdempotencyTable") +table = os.getenv("IDEMPOTENCY_TABLE", "") +dynamodb = DynamoDBPersistenceLayer(table_name=table) config = IdempotencyConfig(event_key_jmespath="order_id") # see Choosing a payload subset section diff --git a/examples/idempotency/src/working_with_idempotent_function_dataclass.py b/examples/idempotency/src/working_with_idempotent_function_dataclass.py index e56c0b42029..3a4e347b22a 100644 --- a/examples/idempotency/src/working_with_idempotent_function_dataclass.py +++ b/examples/idempotency/src/working_with_idempotent_function_dataclass.py @@ -1,3 +1,4 @@ +import os from dataclasses import dataclass from aws_lambda_powertools.utilities.idempotency import ( @@ -7,7 +8,8 @@ ) from aws_lambda_powertools.utilities.typing import LambdaContext -dynamodb = DynamoDBPersistenceLayer(table_name="IdempotencyTable") +table = os.getenv("IDEMPOTENCY_TABLE", "") +dynamodb = DynamoDBPersistenceLayer(table_name=table) config = IdempotencyConfig(event_key_jmespath="order_id") # see Choosing a payload subset section @@ -24,12 +26,14 @@ class Order: @idempotent_function(data_keyword_argument="order", config=config, persistence_store=dynamodb) -def process_order(order: Order): +def process_order(order: Order): # (1)! return f"processed order {order.order_id}" def lambda_handler(event: dict, context: LambdaContext): - config.register_lambda_context(context) # see Lambda timeouts section + # see Lambda timeouts section + config.register_lambda_context(context) # (2)! + order_item = OrderItem(sku="fake", description="sample") order = Order(item=order_item, order_id=1) diff --git a/examples/idempotency/src/working_with_idempotent_function_pydantic.py b/examples/idempotency/src/working_with_idempotent_function_pydantic.py index 5dfd42ae0a8..45b57499a29 100644 --- a/examples/idempotency/src/working_with_idempotent_function_pydantic.py +++ b/examples/idempotency/src/working_with_idempotent_function_pydantic.py @@ -1,3 +1,5 @@ +import os + from aws_lambda_powertools.utilities.idempotency import ( DynamoDBPersistenceLayer, IdempotencyConfig, @@ -6,7 +8,8 @@ from aws_lambda_powertools.utilities.parser import BaseModel from aws_lambda_powertools.utilities.typing import LambdaContext -dynamodb = DynamoDBPersistenceLayer(table_name="IdempotencyTable") +table = os.getenv("IDEMPOTENCY_TABLE", "") +dynamodb = DynamoDBPersistenceLayer(table_name=table) config = IdempotencyConfig(event_key_jmespath="order_id") # see Choosing a payload subset section diff --git a/examples/idempotency/src/working_with_lambda_timeout.py b/examples/idempotency/src/working_with_lambda_timeout.py index 82b8130b6b7..eac423607ad 100644 --- a/examples/idempotency/src/working_with_lambda_timeout.py +++ b/examples/idempotency/src/working_with_lambda_timeout.py @@ -1,3 +1,5 @@ +import os + from aws_lambda_powertools.utilities.data_classes.sqs_event import SQSRecord from aws_lambda_powertools.utilities.idempotency import ( DynamoDBPersistenceLayer, @@ -6,7 +8,8 @@ ) from aws_lambda_powertools.utilities.typing import LambdaContext -persistence_layer = DynamoDBPersistenceLayer(table_name="IdempotencyTable") +table = os.getenv("IDEMPOTENCY_TABLE", "") +persistence_layer = DynamoDBPersistenceLayer(table_name=table) config = IdempotencyConfig() diff --git a/examples/idempotency/src/working_with_local_cache.py b/examples/idempotency/src/working_with_local_cache.py index 82f39dff2ef..571098715f7 100644 --- a/examples/idempotency/src/working_with_local_cache.py +++ b/examples/idempotency/src/working_with_local_cache.py @@ -1,3 +1,5 @@ +import os + from aws_lambda_powertools.utilities.idempotency import ( DynamoDBPersistenceLayer, IdempotencyConfig, @@ -5,10 +7,12 @@ ) from aws_lambda_powertools.utilities.typing import LambdaContext -persistence_layer = DynamoDBPersistenceLayer(table_name="IdempotencyTable") +table = os.getenv("IDEMPOTENCY_TABLE", "") +persistence_layer = DynamoDBPersistenceLayer(table_name=table) config = IdempotencyConfig( - event_key_jmespath="body", - use_local_cache=True, + event_key_jmespath="powertools_json(body)", + # by default, it holds 256 items in a Least-Recently-Used (LRU) manner + use_local_cache=True, # (1)! ) diff --git a/examples/idempotency/src/working_with_payload_subset.py b/examples/idempotency/src/working_with_payload_subset.py index 9fcc828fe1d..c16508cbbb2 100644 --- a/examples/idempotency/src/working_with_payload_subset.py +++ b/examples/idempotency/src/working_with_payload_subset.py @@ -1,4 +1,5 @@ import json +import os from dataclasses import dataclass, field from uuid import uuid4 @@ -9,7 +10,8 @@ ) from aws_lambda_powertools.utilities.typing import LambdaContext -persistence_layer = DynamoDBPersistenceLayer(table_name="IdempotencyTable") +table = os.getenv("IDEMPOTENCY_TABLE", "") +persistence_layer = DynamoDBPersistenceLayer(table_name=table) # Deserialize JSON string under the "body" key # then extract "user" and "product_id" data @@ -23,8 +25,7 @@ class Payment: payment_id: str = field(default_factory=lambda: f"{uuid4()}") -class PaymentError(Exception): - ... +class PaymentError(Exception): ... @idempotent(config=config, persistence_store=persistence_layer) diff --git a/examples/idempotency/src/working_with_pydantic_deduced_output_serializer.py b/examples/idempotency/src/working_with_pydantic_deduced_output_serializer.py index f24fda81e86..b904a5ad670 100644 --- a/examples/idempotency/src/working_with_pydantic_deduced_output_serializer.py +++ b/examples/idempotency/src/working_with_pydantic_deduced_output_serializer.py @@ -1,3 +1,5 @@ +import os + from aws_lambda_powertools.utilities.idempotency import ( DynamoDBPersistenceLayer, IdempotencyConfig, @@ -7,7 +9,8 @@ from aws_lambda_powertools.utilities.parser import BaseModel from aws_lambda_powertools.utilities.typing import LambdaContext -dynamodb = DynamoDBPersistenceLayer(table_name="IdempotencyTable") +table = os.getenv("IDEMPOTENCY_TABLE", "") +dynamodb = DynamoDBPersistenceLayer(table_name=table) config = IdempotencyConfig(event_key_jmespath="order_id") # see Choosing a payload subset section diff --git a/examples/idempotency/src/working_with_pydantic_explicitly_output_serializer.py b/examples/idempotency/src/working_with_pydantic_explicitly_output_serializer.py index 7bd63dfcd9f..b888b58a87c 100644 --- a/examples/idempotency/src/working_with_pydantic_explicitly_output_serializer.py +++ b/examples/idempotency/src/working_with_pydantic_explicitly_output_serializer.py @@ -1,3 +1,5 @@ +import os + from aws_lambda_powertools.utilities.idempotency import ( DynamoDBPersistenceLayer, IdempotencyConfig, @@ -7,7 +9,8 @@ from aws_lambda_powertools.utilities.parser import BaseModel from aws_lambda_powertools.utilities.typing import LambdaContext -dynamodb = DynamoDBPersistenceLayer(table_name="IdempotencyTable") +table = os.getenv("IDEMPOTENCY_TABLE", "") +dynamodb = DynamoDBPersistenceLayer(table_name=table) config = IdempotencyConfig(event_key_jmespath="order_id") # see Choosing a payload subset section diff --git a/examples/idempotency/src/working_with_record_expiration.py b/examples/idempotency/src/working_with_record_expiration.py index 738b4749ebc..e1696ee7bbf 100644 --- a/examples/idempotency/src/working_with_record_expiration.py +++ b/examples/idempotency/src/working_with_record_expiration.py @@ -1,3 +1,5 @@ +import os + from aws_lambda_powertools.utilities.idempotency import ( DynamoDBPersistenceLayer, IdempotencyConfig, @@ -5,10 +7,11 @@ ) from aws_lambda_powertools.utilities.typing import LambdaContext -persistence_layer = DynamoDBPersistenceLayer(table_name="IdempotencyTable") +table = os.getenv("IDEMPOTENCY_TABLE", "") +persistence_layer = DynamoDBPersistenceLayer(table_name=table) config = IdempotencyConfig( event_key_jmespath="body", - expires_after_seconds=5 * 60, # 5 minutes + expires_after_seconds=24 * 60 * 60, # 24 hours ) diff --git a/examples/idempotency/src/working_with_response_hook.py b/examples/idempotency/src/working_with_response_hook.py index 2c2208d25a5..e800f3a0356 100644 --- a/examples/idempotency/src/working_with_response_hook.py +++ b/examples/idempotency/src/working_with_response_hook.py @@ -1,4 +1,5 @@ import datetime +import os import uuid from typing import Dict @@ -30,7 +31,8 @@ def my_response_hook(response: Dict, idempotent_data: DataRecord) -> Dict: return response -dynamodb = DynamoDBPersistenceLayer(table_name="IdempotencyTable") +table = os.getenv("IDEMPOTENCY_TABLE", "") +dynamodb = DynamoDBPersistenceLayer(table_name=table) config = IdempotencyConfig(response_hook=my_response_hook) diff --git a/examples/idempotency/src/working_with_validation_payload.py b/examples/idempotency/src/working_with_validation_payload.py index d81e7d183bd..12b8423e7c4 100644 --- a/examples/idempotency/src/working_with_validation_payload.py +++ b/examples/idempotency/src/working_with_validation_payload.py @@ -1,15 +1,24 @@ +import os from dataclasses import dataclass, field from uuid import uuid4 +from aws_lambda_powertools import Logger from aws_lambda_powertools.utilities.idempotency import ( DynamoDBPersistenceLayer, IdempotencyConfig, idempotent, ) +from aws_lambda_powertools.utilities.idempotency.exceptions import IdempotencyValidationError from aws_lambda_powertools.utilities.typing import LambdaContext -persistence_layer = DynamoDBPersistenceLayer(table_name="IdempotencyTable") -config = IdempotencyConfig(event_key_jmespath='["user_id", "product_id"]', payload_validation_jmespath="amount") +logger = Logger() + +table = os.getenv("IDEMPOTENCY_TABLE", "") +persistence_layer = DynamoDBPersistenceLayer(table_name=table) +config = IdempotencyConfig( + event_key_jmespath='["user_id", "product_id"]', + payload_validation_jmespath="amount", +) @dataclass @@ -21,8 +30,7 @@ class Payment: payment_id: str = field(default_factory=lambda: f"{uuid4()}") -class PaymentError(Exception): - ... +class PaymentError(Exception): ... @idempotent(config=config, persistence_store=persistence_layer) @@ -34,6 +42,12 @@ def lambda_handler(event: dict, context: LambdaContext): "message": "success", "statusCode": 200, } + except IdempotencyValidationError: + logger.exception("Payload tampering detected", payment=payment, failure_type="validation") + return { + "message": "Unable to process payment at this time. Try again later.", + "statusCode": 500, + } except Exception as exc: raise PaymentError(f"Error creating payment {str(exc)}") diff --git a/examples/idempotency/templates/cfn_redis_serverless.yaml b/examples/idempotency/templates/cfn_redis_serverless.yaml index 9087efce6f9..8ce9d67f3cb 100644 --- a/examples/idempotency/templates/cfn_redis_serverless.yaml +++ b/examples/idempotency/templates/cfn_redis_serverless.yaml @@ -1,4 +1,5 @@ -AWSTemplateFormatVersion: '2010-09-09' +AWSTemplateFormatVersion: "2010-09-09" +Transform: AWS::Serverless-2016-10-31 Resources: RedisServerlessIdempotency: @@ -7,7 +8,24 @@ Resources: Engine: redis ServerlessCacheName: redis-cache SecurityGroupIds: # (1)! - - security-{your_sg_id} + - security-{your_sg_id} SubnetIds: + - subnet-{your_subnet_id_1} + - subnet-{your_subnet_id_2} + + HelloWorldFunction: + Type: AWS::Serverless::Function + Properties: + Runtime: python3.12 + Handler: app.py + VpcConfig: # (1)! + SecurityGroupIds: + - security-{your_sg_id} + SubnetIds: - subnet-{your_subnet_id_1} - subnet-{your_subnet_id_2} + Environment: + Variables: + POWERTOOLS_SERVICE_NAME: sample + REDIS_HOST: !GetAtt RedisServerlessIdempotency.Endpoint.Address + REDIS_PORT: !GetAtt RedisServerlessIdempotency.Endpoint.Port diff --git a/examples/idempotency/templates/sam.yaml b/examples/idempotency/templates/sam.yaml index c4eaf766c23..4faab5c4225 100644 --- a/examples/idempotency/templates/sam.yaml +++ b/examples/idempotency/templates/sam.yaml @@ -21,11 +21,14 @@ Resources: Handler: app.py Policies: - Statement: - - Sid: AllowDynamodbReadWrite - Effect: Allow - Action: - - dynamodb:PutItem - - dynamodb:GetItem - - dynamodb:UpdateItem - - dynamodb:DeleteItem - Resource: !GetAtt IdempotencyTable.Arn + - Sid: AllowDynamodbReadWrite + Effect: Allow + Action: + - dynamodb:PutItem + - dynamodb:GetItem + - dynamodb:UpdateItem + - dynamodb:DeleteItem + Resource: !GetAtt IdempotencyTable.Arn + Environment: + Variables: + IDEMPOTENCY_TABLE: !Ref IdempotencyTable diff --git a/examples/idempotency/templates/sam_redis_vpc.yaml b/examples/idempotency/templates/sam_redis_vpc.yaml deleted file mode 100644 index 921b1e75b84..00000000000 --- a/examples/idempotency/templates/sam_redis_vpc.yaml +++ /dev/null @@ -1,14 +0,0 @@ -AWSTemplateFormatVersion: '2010-09-09' -Transform: AWS::Serverless-2016-10-31 -Resources: - HelloWorldFunction: - Type: AWS::Serverless::Function - Properties: - Runtime: python3.11 - Handler: app.py - VpcConfig: # (1)! - SecurityGroupIds: - - security-{your_sg_id} - SubnetIds: - - subnet-{your_subnet_id_1} - - subnet-{your_subnet_id_2} diff --git a/includes/abbreviations.md b/includes/abbreviations.md index ed52b93fe64..5e0db4dcb27 100644 --- a/includes/abbreviations.md +++ b/includes/abbreviations.md @@ -1 +1,2 @@ *[observability provider]: An AWS Lambda Observability Partner +*[unhandled exception]: An exception that is not caught by any explicit try/except block diff --git a/mkdocs.yml b/mkdocs.yml index d2bab86cd22..988f7dc5f06 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -147,3 +147,13 @@ extra: version: provider: mike default: latest + social: + - icon: fontawesome/brands/discord + link: https://discord.gg/B8zZKbbyET + name: Discord Server for Powertools for AWS + - icon: material/web + link: https://powertools.aws.dev/ + name: Official website for Powertools for AWS + - icon: simple/python + link: https://pypi.org/project/aws-lambda-powertools/ + name: PyPi package