title	description
Idempotency	Utility

The idempotency utility provides a simple solution to convert your Lambda functions into idempotent operations which are safe to retry.

Key features

Prevent Lambda handler from executing more than once on the same event payload during a time window
Ensure Lambda handler returns the same result when called with the same payload
Select a subset of the event as the idempotency key using JMESPath expressions
Set a time window in which records with the same payload should be considered duplicates
Expires in-progress executions if the Lambda function times out halfway through
Support Amazon DynamoDB and Redis as persistence layers

Terminology

The property of idempotency means that an operation does not cause additional side effects if it is called more than once with the same input parameters.

Idempotent operations will return the same result when they are called multiple times with the same parameters. This makes idempotent operations safe to retry.

Idempotency key is a hash representation of either the entire event or a specific configured subset of the event, and invocation results are JSON serialized and stored in your persistence storage layer.

Idempotency record is the data representation of an idempotent request saved in your preferred storage layer. We use it to coordinate whether a request is idempotent, whether it's still valid or expired based on timestamps, etc.

```mermaid classDiagram direction LR class IdempotencyRecord { idempotency_key str status Status expiry_timestamp int in_progress_expiry_timestamp int response_data Json~str~ payload_hash str } class Status { <> INPROGRESS COMPLETE EXPIRED internal_only } IdempotencyRecord -- Status ```

Idempotency record representation

Getting started

???+ note This section uses DynamoDB as the default idempotent persistence storage layer. If you are interested in using Redis as the persistence storage layer, check out the Redis as persistence storage layer Section.

IAM Permissions

Your Lambda function IAM Role must have dynamodb:GetItem, dynamodb:PutItem, dynamodb:UpdateItem and dynamodb:DeleteItem IAM permissions before using this feature.

???+ note If you're using our example AWS Serverless Application Model (SAM), AWS Cloud Development Kit (CDK), or Terraform it already adds the required permissions.

Required resources

Before getting started, you need to create a persistent storage layer where the idempotency utility can store its state - your lambda functions will need read and write access to it.

We currently support Amazon DynamoDB and Redis as a storage layer. The following example demonstrates how to create a table in DynamoDB. If you prefer to use Redis, refer go to the section RedisPersistenceLayer section.

Default table configuration

If you're not changing the default configuration for the DynamoDB persistence layer, this is the expected default configuration:

Configuration	Value	Notes
Partition key	`id`
TTL attribute name	`expiration`	This can only be configured after your table is created if you're using AWS Console

???+ tip "Tip: You can share a single state table for all functions" You can reuse the same DynamoDB table to store idempotency state. We add module_name and qualified name for classes and functions{target="_blank" rel="nofollow"} in addition to the idempotency key as a hash key.

=== "AWS Serverless Application Model (SAM) example"

```yaml hl_lines="6-14 24-31"
--8<-- "examples/idempotency/templates/sam.yaml"
```

=== "AWS Cloud Development Kit (CDK)"

```python hl_lines="10 13 16 19-21"
--8<-- "examples/idempotency/templates/cdk.py"
```

=== "Terraform"

```terraform hl_lines="14-26 64-70"
--8<-- "examples/idempotency/templates/terraform.tf"
```

???+ warning "Warning: Large responses with DynamoDB persistence layer" When using this utility with DynamoDB, your function's responses must be smaller than 400KB{target="_blank"}.

Larger items cannot be written to DynamoDB and will cause exceptions. If your response exceeds 400kb, consider using Redis as your persistence layer.

???+ info "Info: DynamoDB"

During the first invocation with a payload, the Lambda function executes both a `PutItem` and an `UpdateItem` operations to store the data in DynamoDB. If the result returned by your Lambda is less than 1kb, you can expect 2 WCUs per Lambda invocation.

On subsequent invocations with the same payload, you can expect just 1 `PutItem` request to DynamoDB.

**Note:** While we try to minimize requests to DynamoDB to 1 per invocation, if your boto3 version is lower than `1.26.194`, you may experience 2 requests in every invocation. Ensure to check your boto3 version and review the [DynamoDB pricing documentation](https://aws.amazon.com/dynamodb/pricing/){target="_blank"} to estimate the cost.

Idempotent decorator

You can quickly start by initializing the DynamoDBPersistenceLayer class and using it with the idempotent decorator on your lambda handler.

???+ note In this example, the entire Lambda handler is treated as a single idempotent operation. If your Lambda handler can cause multiple side effects, or you're only interested in making a specific logic idempotent, use idempotent_function instead.

!!! tip "See Choosing a payload subset for idempotency for more elaborate use cases."

=== "Idempotent decorator"

```python hl_lines="4-7 10 24"
--8<-- "examples/idempotency/src/getting_started_with_idempotency.py"
```

=== "Sample event"

```json
--8<-- "examples/idempotency/src/getting_started_with_idempotency_payload.json"
```

After processing this request successfully, a second request containing the exact same payload above will now return the same response, ensuring our customer isn't charged twice.

!!! question "New to idempotency concept? Please review our Terminology section if you haven't yet."

Idempotent_function decorator

Similar to idempotent decorator, you can use idempotent_function decorator for any synchronous Python function.

When using idempotent_function, you must tell us which keyword parameter in your function signature has the data we should use via data_keyword_argument.

!!! tip "We support JSON serializable data, Python Dataclasses{target="_blank" rel="nofollow"}, Parser/Pydantic Models{target="_blank"}, and our Event Source Data Classes{target="_blank"}."

???+ warning "Limitation" Make sure to call your decorated function using keyword arguments.

=== "Using Dataclasses"

```python hl_lines="3-7 11 26 37"
--8<-- "examples/idempotency/src/working_with_idempotent_function_dataclass.py"
```

=== "Using Pydantic"

```python hl_lines="1-5 10 23 34"
--8<-- "examples/idempotency/src/working_with_idempotent_function_pydantic.py"
```

Output serialization

By default, idempotent_function serializes, stores, and returns your annotated function's result as a JSON object. You can change this behavior using output_serializer parameter.

The output serializer supports any JSON serializable data, Python Dataclasses and Pydantic Models.

!!! info "When using the output_serializer parameter, the data will continue to be stored in DynamoDB as a JSON object."

=== "Pydantic"

You can use `PydanticSerializer` to automatically serialize what's retrieved from the persistent storage based on the return type annotated.

=== "Inferring via the return type"

    ```python hl_lines="6 24 25 32 36 45"
    --8<-- "examples/idempotency/src/working_with_pydantic_deduced_output_serializer.py"
    ```

    1. We'll use `OrderOutput` to instantiate a new object using the data retrieved from persistent storage as input. <br><br> This ensures the return of the function is not impacted when `@idempotent_function` is used.

=== "Explicit model type"

    Alternatively, you can provide an explicit model as an input to `PydanticSerializer`.

    ```python hl_lines="6 24 25 32 35 44"
    --8<-- "examples/idempotency/src/working_with_pydantic_explicitly_output_serializer.py"
    ```

=== "Dataclasses"

 You can use `DataclassSerializer` to automatically serialize what's retrieved from the persistent storage based on the return type annotated.

=== "Inferring via the return type"

    ```python hl_lines="8 27-29 36 40 49"
    --8<-- "examples/idempotency/src/working_with_dataclass_deduced_output_serializer.py"
    ```

    1. We'll use `OrderOutput` to instantiate a new object using the data retrieved from persistent storage as input. <br><br> This ensures the return of the function is not impacted when `@idempotent_function` is used.

=== "Explicit model type"

    Alternatively, you can provide an explicit model as an input to `DataclassSerializer`.

    ```python hl_lines="8 27-29 36 39 48"
    --8<-- "examples/idempotency/src/working_with_dataclass_explicitly_output_serializer.py"
    ```

=== "Any type"

You can use `CustomDictSerializer` to have full control over the serialization process for any type. It expects two functions:

* **to_dict**. Function to convert any type to a JSON serializable dictionary before it saves into the persistent storage.
* **from_dict**. Function to convert from a dictionary retrieved from persistent storage and serialize in its original form.

```python hl_lines="8 32 36 40 50 53"
--8<-- "examples/idempotency/src/working_with_idempotent_function_custom_output_serializer.py"
```

1. This function does the following <br><br>**1**. Receives the return from `process_order`<br> **2**. Converts to dictionary before it can be saved into the persistent storage.
2. This function does the following <br><br>**1**. Receives the dictionary saved into the persistent storage <br>**1** Serializes to `OrderOutput` before `@idempotent` returns back to the caller.
3. This serializer receives both functions so it knows who to call when to serialize to and from dictionary.

Batch integration

You can can easily integrate with Batch utility{target="_blank"} via context manager. This ensures that you process each record in an idempotent manner, and guard against a Lambda timeout idempotent situation.

???+ "Choosing an unique batch record attribute" In this example, we choose messageId as our idempotency key since we know it'll be unique.

Depending on your use case, it might be more accurate [to choose another field](#choosing-a-payload-subset-for-idempotency) your producer intentionally set to define uniqueness.

=== "Integration with Batch Processor"

```python hl_lines="2 12 16 20 31 35 37"
--8<-- "examples/idempotency/src/integrate_idempotency_with_batch_processor.py"
```

=== "Sample event"

```json hl_lines="4"
--8<-- "examples/idempotency/src/integrate_idempotency_with_batch_processor_payload.json"
```

Choosing a payload subset for idempotency

???+ tip "Tip: Dealing with always changing payloads" When dealing with a more elaborate payload, where parts of the payload always change, you should use event_key_jmespath parameter.

Use IdempotencyConfig to instruct the idempotent decorator to only use a portion of your payload to verify whether a request is idempotent, and therefore it should not be retried.

Payment scenario

In this example, we have a Lambda handler that creates a payment for a user subscribing to a product. We want to ensure that we don't accidentally charge our customer by subscribing them more than once.

Imagine the function executes successfully, but the client never receives the response due to a connection issue. It is safe to retry in this instance, as the idempotent decorator will return a previously saved response.

What we want here is to instruct Idempotency to use user_id and product_id fields from our incoming payload as our idempotency key. If we were to treat the entire request as our idempotency key, a simple HTTP header change would cause our customer to be charged twice.

???+ tip "Deserializing JSON strings in payloads for increased accuracy." The payload extracted by the event_key_jmespath is treated as a string by default. This means there could be differences in whitespace even when the JSON payload itself is identical.

To alter this behaviour, we can use the [JMESPath built-in function](jmespath_functions.md#powertools_json-function){target="_blank"} `powertools_json()` to treat the payload as a JSON object (dict) rather than a string.

=== "Payment function"

```python hl_lines="5-9 16 30"
--8<-- "examples/idempotency/src/working_with_payload_subset.py"
```

=== "Sample event"

```json hl_lines="28"
--8<-- "examples/idempotency/src/working_with_payload_subset_payload.json"
```

Lambda timeouts

???+ note This is automatically done when you decorate your Lambda handler with @idempotent decorator.

To prevent against extended failed retries when a Lambda function times out{target="_blank"}, Powertools for AWS Lambda (Python) calculates and includes the remaining invocation available time as part of the idempotency record.

???+ example If a second invocation happens after this timestamp, and the record is marked as INPROGRESS, we will execute the invocation again as if it was in the EXPIRED state (e.g, expire_seconds field elapsed).

This means that if an invocation expired during execution, it will be quickly executed again on the next retry.

???+ important If you are only using the @idempotent_function decorator to guard isolated parts of your code, you must use register_lambda_context available in the idempotency config object to benefit from this protection.

Here is an example on how you register the Lambda context in your handler:

=== "Registering the Lambda context"

```python hl_lines="11 20"
--8<-- "examples/idempotency/src/working_with_lambda_timeout.py"
```

Handling exceptions

If you are using the idempotent decorator on your Lambda handler, any unhandled exceptions that are raised during the code execution will cause the record in the persistence layer to be deleted. This means that new invocations will execute your code again despite having the same payload. If you don't want the record to be deleted, you need to catch exceptions within the idempotent function and return a successful response.

```mermaid sequenceDiagram participant Client participant Lambda participant Persistence Layer Client->>Lambda: Invoke (event) Lambda->>Persistence Layer: Get or set (id=event.search(payload)) activate Persistence Layer Note right of Persistence Layer: Locked during this time. Prevents multiple
Lambda invocations with the same
payload running concurrently. Lambda--xLambda: Call handler (event).
Raises exception Lambda->>Persistence Layer: Delete record (id=event.search(payload)) deactivate Persistence Layer Lambda-->>Client: Return error response ``` Idempotent sequence exception

If you are using idempotent_function, any unhandled exceptions that are raised inside the decorated function will cause the record in the persistence layer to be deleted, and allow the function to be executed again if retried.

If an Exception is raised outside the scope of the decorated function and after your function has been called, the persistent record will not be affected. In this case, idempotency will be maintained for your decorated function. Example:

=== "Handling exceptions"

```python hl_lines="18-22 28 31"
--8<-- "examples/idempotency/src/working_with_exceptions.py"
```

???+ warning We will raise IdempotencyPersistenceLayerError if any of the calls to the persistence layer fail unexpectedly.

As this happens outside the scope of your decorated function, you are not able to catch it if you're using the `idempotent` decorator on your Lambda handler.

Persistence layers

DynamoDBPersistenceLayer

This persistence layer is built-in, allowing you to use an existing DynamoDB table or create a new one dedicated to idempotency state (recommended).

=== "Customizing DynamoDBPersistenceLayer to suit your table structure"

```python hl_lines="7-15"
--8<-- "examples/idempotency/src/customize_persistence_layer.py"
```

When using DynamoDB as the persistence layer, you can customize the attribute names by passing the following parameters during the initialization of the persistence layer:

Parameter	Required	Default	Description
table_name	✔️		Table name to store state
key_attr		`id`	Partition key of the table. Hashed representation of the payload (unless sort_key_attr is specified)
expiry_attr		`expiration`	Unix timestamp of when record expires
in_progress_expiry_attr		`in_progress_expiration`	Unix timestamp of when record expires while in progress (in case of the invocation times out)
status_attr		`status`	Stores status of the lambda execution during and after invocation
data_attr		`data`	Stores results of successfully executed Lambda handlers
validation_key_attr		`validation`	Hashed representation of the parts of the event used for validation
sort_key_attr			Sort key of the table (if table is configured with a sort key).
static_pk_value		`idempotency#{LAMBDA_FUNCTION_NAME}`	Static value to use as the partition key. Only used when sort_key_attr is set.

RedisPersistenceLayer

This persistence layer is built-in, allowing you to use an existing Redis service. For optimal performance and compatibility, it is strongly recommended to use a Redis service version 7 or higher.

=== "Customizing RedisPersistenceLayer to suit your data structure"

```python hl_lines="9-16"
--8<-- "examples/idempotency/src/customize_persistence_layer_redis.py"
```

When using Redis as the persistence layer, you can customize the attribute names by providing the following parameters upon initialization of the persistence layer:

Parameter	Default	Description
in_progress_expiry_attr	`in_progress_expiration`	Unix timestamp of when record expires while in progress (in case of the invocation times out)
status_attr	`status`	Stores status of the Lambda execution during and after invocation
data_attr	`data`	Stores results of successfully executed Lambda handlers
validation_key_attr	`validation`	Hashed representation of the parts of the event used for validation

Idempotency request flow

The following sequence diagrams explain how the Idempotency feature behaves under different scenarios.

Successful request

```mermaid sequenceDiagram participant Client participant Lambda participant Persistence Layer alt initial request Client->>Lambda: Invoke (event) Lambda->>Persistence Layer: Get or set idempotency_key=hash(payload) activate Persistence Layer Note over Lambda,Persistence Layer: Set record status to INPROGRESS.
Prevents concurrent invocations
with the same payload Lambda-->>Lambda: Call your function Lambda->>Persistence Layer: Update record with result deactivate Persistence Layer Persistence Layer-->>Persistence Layer: Update record Note over Lambda,Persistence Layer: Set record status to COMPLETE.
New invocations with the same payload
now return the same result Lambda-->>Client: Response sent to client else retried request Client->>Lambda: Invoke (event) Lambda->>Persistence Layer: Get or set idempotency_key=hash(payload) activate Persistence Layer Persistence Layer-->>Lambda: Already exists in persistence layer. deactivate Persistence Layer Note over Lambda,Persistence Layer: Record status is COMPLETE and not expired Lambda-->>Client: Same response sent to client end ``` Idempotent successful request

Successful request with cache enabled

!!! note "In-memory cache is disabled by default."

```mermaid sequenceDiagram participant Client participant Lambda participant Persistence Layer alt initial request Client->>Lambda: Invoke (event) Lambda->>Persistence Layer: Get or set idempotency_key=hash(payload) activate Persistence Layer Note over Lambda,Persistence Layer: Set record status to INPROGRESS.
Prevents concurrent invocations
with the same payload Lambda-->>Lambda: Call your function Lambda->>Persistence Layer: Update record with result deactivate Persistence Layer Persistence Layer-->>Persistence Layer: Update record Note over Lambda,Persistence Layer: Set record status to COMPLETE.
New invocations with the same payload
now return the same result Lambda-->>Lambda: Save record and result in memory Lambda-->>Client: Response sent to client else retried request Client->>Lambda: Invoke (event) Lambda-->>Lambda: Get idempotency_key=hash(payload) Note over Lambda,Persistence Layer: Record status is COMPLETE and not expired Lambda-->>Client: Same response sent to client end ``` Idempotent successful request cached

Successful request with response_hook configured

```mermaid sequenceDiagram participant Client participant Lambda participant Response hook participant Persistence Layer alt initial request Client->>Lambda: Invoke (event) Lambda->>Persistence Layer: Get or set idempotency_key=hash(payload) activate Persistence Layer Note over Lambda,Persistence Layer: Set record status to INPROGRESS.
Prevents concurrent invocations
with the same payload Lambda-->>Lambda: Call your function Lambda->>Persistence Layer: Update record with result deactivate Persistence Layer Persistence Layer-->>Persistence Layer: Update record Note over Lambda,Persistence Layer: Set record status to COMPLETE.
New invocations with the same payload
now return the same result Lambda-->>Client: Response sent to client else retried request Client->>Lambda: Invoke (event) Lambda->>Persistence Layer: Get or set idempotency_key=hash(payload) activate Persistence Layer Persistence Layer-->>Response hook: Already exists in persistence layer. deactivate Persistence Layer Note over Response hook,Persistence Layer: Record status is COMPLETE and not expired Response hook->>Lambda: Response hook invoked Lambda-->>Client: Manipulated idempotent response sent to client end ``` Successful idempotent request with a response hook

Expired idempotency records

```mermaid sequenceDiagram participant Client participant Lambda participant Persistence Layer alt initial request Client->>Lambda: Invoke (event) Lambda->>Persistence Layer: Get or set idempotency_key=hash(payload) activate Persistence Layer Note over Lambda,Persistence Layer: Set record status to INPROGRESS.
Prevents concurrent invocations
with the same payload Lambda-->>Lambda: Call your function Lambda->>Persistence Layer: Update record with result deactivate Persistence Layer Persistence Layer-->>Persistence Layer: Update record Note over Lambda,Persistence Layer: Set record status to COMPLETE.
New invocations with the same payload
now return the same result Lambda-->>Client: Response sent to client else retried request Client->>Lambda: Invoke (event) Lambda->>Persistence Layer: Get or set idempotency_key=hash(payload) activate Persistence Layer Persistence Layer-->>Lambda: Already exists in persistence layer. deactivate Persistence Layer Note over Lambda,Persistence Layer: Record status is COMPLETE but expired hours ago loop Repeat initial request process Note over Lambda,Persistence Layer: 1. Set record to INPROGRESS,
2. Call your function,
3. Set record to COMPLETE end Lambda-->>Client: Same response sent to client end ``` Previous Idempotent request expired

Concurrent identical in-flight requests

```mermaid sequenceDiagram participant Client participant Lambda participant Persistence Layer Client->>Lambda: Invoke (event) Lambda->>Persistence Layer: Get or set idempotency_key=hash(payload) activate Persistence Layer Note over Lambda,Persistence Layer: Set record status to INPROGRESS.
Prevents concurrent invocations
with the same payload par Second request Client->>Lambda: Invoke (event) Lambda->>Persistence Layer: Get or set idempotency_key=hash(payload) Lambda--xLambda: IdempotencyAlreadyInProgressError Lambda->>Client: Error sent to client if unhandled end Lambda-->>Lambda: Call your function Lambda->>Persistence Layer: Update record with result deactivate Persistence Layer Persistence Layer-->>Persistence Layer: Update record Note over Lambda,Persistence Layer: Set record status to COMPLETE.
New invocations with the same payload
now return the same result Lambda-->>Client: Response sent to client ``` Concurrent identical in-flight requests

Lambda request timeout

```mermaid sequenceDiagram participant Client participant Lambda participant Persistence Layer alt initial request Client->>Lambda: Invoke (event) Lambda->>Persistence Layer: Get or set idempotency_key=hash(payload) activate Persistence Layer Note over Lambda,Persistence Layer: Set record status to INPROGRESS.
Prevents concurrent invocations
with the same payload Lambda-->>Lambda: Call your function Note right of Lambda: Time out Lambda--xLambda: Time out error Lambda-->>Client: Return error response deactivate Persistence Layer else retry after Lambda timeout elapses Client->>Lambda: Invoke (event) Lambda->>Persistence Layer: Get or set idempotency_key=hash(payload) activate Persistence Layer Note over Lambda,Persistence Layer: Set record status to INPROGRESS.
Reset in_progress_expiry attribute Lambda-->>Lambda: Call your function Lambda->>Persistence Layer: Update record with result deactivate Persistence Layer Persistence Layer-->>Persistence Layer: Update record Lambda-->>Client: Response sent to client end ``` Idempotent request during and after Lambda timeouts

Optional idempotency key

```mermaid sequenceDiagram participant Client participant Lambda participant Persistence Layer alt request with idempotency key Client->>Lambda: Invoke (event) Lambda->>Persistence Layer: Get or set idempotency_key=hash(payload) activate Persistence Layer Note over Lambda,Persistence Layer: Set record status to INPROGRESS.
Prevents concurrent invocations
with the same payload Lambda-->>Lambda: Call your function Lambda->>Persistence Layer: Update record with result deactivate Persistence Layer Persistence Layer-->>Persistence Layer: Update record Note over Lambda,Persistence Layer: Set record status to COMPLETE.
New invocations with the same payload
now return the same result Lambda-->>Client: Response sent to client else request(s) without idempotency key Client->>Lambda: Invoke (event) Note over Lambda: Idempotency key is missing Note over Persistence Layer: Skips any operation to fetch, update, and delete Lambda-->>Lambda: Call your function Lambda-->>Client: Response sent to client end ``` Optional idempotency key

Race condition with Redis

```mermaid graph TD; A(Existing orphan record in redis)-->A1; A1[Two Lambda invoke at same time]-->B1[Lambda handler1]; B1-->B2[Fetch from Redis]; B2-->B3[Handler1 got orphan record]; B3-->B4[Handler1 acquired lock]; B4-->B5[Handler1 overwrite orphan record] B5-->B6[Handler1 continue to execution]; A1-->C1[Lambda handler2]; C1-->C2[Fetch from Redis]; C2-->C3[Handler2 got orphan record]; C3-->C4[Handler2 failed to acquire lock]; C4-->C5[Handler2 wait and fetch from Redis]; C5-->C6[Handler2 return without executing]; B6-->D(Lambda handler executed only once); C6-->D; ``` Race condition with Redis

Redis as persistent storage layer provider

Redis resources

Before setting up Redis as the persistent storage layer provider, you must have an existing Redis service. We recommend you to use Redis compatible services such as Amazon ElastiCache for Redis{target="_blank"} or Amazon MemoryDB for Redis{target="_blank"} as your persistent storage layer provider.

???+ tip "No existing Redis service?" If you don't have an existing Redis service, we recommend using DynamoDB as the persistent storage layer provider.

=== "AWS CloudFormation example"

```yaml hl_lines="5"
--8<-- "examples/idempotency/templates/cfn_redis_serverless.yaml"
```

1. Replace the Security Group ID and Subnet ID to match your VPC settings.

VPC Access

Your Lambda Function must have network access to the Redis endpoint before using it as the idempotency persistent storage layer. In most cases, you will need to configure VPC access{target="_blank"} for your Lambda Function.

???+ tip "Amazon ElastiCache/MemoryDB for Redis as persistent storage layer provider" If you plan to use Amazon ElastiCache for Redis as the idempotency persistent storage layer, you may find this AWS tutorial{target="_blank"} helpful. For those using Amazon MemoryDB for Redis, refer to this AWS tutorial{target="_blank"} specifically for the VPC setup guidance.

After completing the VPC setup, you can use the templates provided below to set up Lambda functions with access to VPC internal subnets.

=== "AWS Serverless Application Model (SAM) example"

```yaml hl_lines="9"
--8<-- "examples/idempotency/templates/sam_redis_vpc.yaml"
```

1. Replace the Security Group ID and Subnet ID to match your VPC settings.

Configuring Redis persistence layer

You can quickly get started by initializing the RedisCachePersistenceLayer class and applying the idempotent decorator to your Lambda handler. For a detailed example of using the RedisCachePersistenceLayer, refer to the Persistence layers section.

???+ info We enforce security best practices by using SSL connections in the RedisCachePersistenceLayer; to disable it, set ssl=False

=== "Use Persistence Layer with Redis config variables" python hl_lines="7-9 12 26" --8<-- "examples/idempotency/src/getting_started_with_idempotency_redis_config.py"

=== "Use established Redis Client" python hl_lines="4 9-11 14 22 36" --8<-- "examples/idempotency/src/getting_started_with_idempotency_redis_client.py"

=== "Sample event"

```json
--8<-- "examples/idempotency/src/getting_started_with_idempotency_payload.json"
```

Custom advanced settings

For advanced configurations, such as setting up SSL certificates or customizing parameters like a custom timeout, you can utilize the Redis client to tailor these specific settings to your needs.

=== "Advanced configuration using AWS Secrets" python hl_lines="7-9 11 13 23" --8<-- "examples/idempotency/src/using_redis_client_with_aws_secrets.py"

1. JSON stored:
{
"REDIS_ENDPOINT": "127.0.0.1",
"REDIS_PORT": "6379",
"REDIS_PASSWORD": "redis-secret"
}

=== "Advanced configuration with local certificates" python hl_lines="12 23-25" --8<-- "examples/idempotency/src/using_redis_client_with_local_certs.py"

1. JSON stored:
{
"REDIS_ENDPOINT": "127.0.0.1",
"REDIS_PORT": "6379",
"REDIS_PASSWORD": "redis-secret"
}
2. redis_user.crt file stored in the "certs" directory of your Lambda function
3. redis_user_private.key file stored in the "certs" directory of your Lambda function
4. redis_ca.pem file stored in the "certs" directory of your Lambda function

Advanced

Customizing the default behavior

Idempotent decorator can be further configured with IdempotencyConfig as seen in the previous example. These are the available options for further configuration

Parameter	Default	Description
event_key_jmespath	`""`	JMESPath expression to extract the idempotency key from the event record using built-in functions{target="_blank"}
payload_validation_jmespath	`""`	JMESPath expression to validate whether certain parameters have changed in the event while the event payload
raise_on_no_idempotency_key	`False`	Raise exception if no idempotency key was found in the request
expires_after_seconds	3600	The number of seconds to wait before a record is expired
use_local_cache	`False`	Whether to locally cache idempotency results
local_cache_max_items	256	Max number of items to store in local cache
hash_function	`md5`	Function to use for calculating hashes, as provided by hashlib{target="_blank" rel="nofollow"} in the standard library.
response_hook	`None`	Function to use for processing the stored Idempotent response. This function hook is called when an existing idempotent response is found. See Manipulating The Idempotent Response

Handling concurrent executions with the same payload

This utility will raise an IdempotencyAlreadyInProgressError exception if you receive multiple invocations with the same payload while the first invocation hasn't completed yet.

???+ info If you receive IdempotencyAlreadyInProgressError, you can safely retry the operation.

This is a locking mechanism for correctness. Since we don't know the result from the first invocation yet, we can't safely allow another concurrent execution.

Using in-memory cache

By default, in-memory local caching is disabled, since we don't know how much memory you consume per invocation compared to the maximum configured in your Lambda function.

???+ note "Note: This in-memory cache is local to each Lambda execution environment" This means it will be effective in cases where your function's concurrency is low in comparison to the number of "retry" invocations with the same payload, because cache might be empty.

You can enable in-memory caching with the use_local_cache parameter:

=== "Caching idempotent transactions in-memory to prevent multiple calls to storage"

```python hl_lines="11"
--8<-- "examples/idempotency/src/working_with_local_cache.py"
```

=== "Sample event"

```json
--8<-- "examples/idempotency/src/working_with_local_cache_payload.json"
```

When enabled, the default is to cache a maximum of 256 records in each Lambda execution environment - You can change it with the local_cache_max_items parameter.

Expiring idempotency records

!!! note "By default, we expire idempotency records after an hour (3600 seconds)."

In most cases, it is not desirable to store the idempotency records forever. Rather, you want to guarantee that the same payload won't be executed within a period of time.

You can change this window with the expires_after_seconds parameter:

=== "Adjusting idempotency record expiration"

```python hl_lines="11"
--8<-- "examples/idempotency/src/working_with_record_expiration.py"
```

=== "Sample event"

```json
--8<-- "examples/idempotency/src/working_with_record_expiration_payload.json"
```

This will mark any records older than 5 minutes as expired, and your function will be executed as normal if it is invoked with a matching payload.

???+ important "Idempotency record expiration vs DynamoDB time-to-live (TTL)" DynamoDB TTL is a feature{target="_blank"} to remove items after a certain period of time, it may occur within 48 hours of expiration.

We don't rely on DynamoDB or any persistence storage layer to determine whether a record is expired to avoid eventual inconsistency states.

Instead, Idempotency records saved in the storage layer contain timestamps that can be verified upon retrieval and double checked within Idempotency feature.

**Why?**

A record might still be valid (`COMPLETE`) when we retrieved, but in some rare cases it might expire a second later. A record could also be [cached in memory](#using-in-memory-cache). You might also want to have idempotent transactions that should expire in seconds.

Payload validation

???+ question "Question: What if your function is invoked with the same payload except some outer parameters have changed?" Example: A payment transaction for a given productID was requested twice for the same customer, however the amount to be paid has changed in the second transaction.

By default, we will return the same result as it returned before, however in this instance it may be misleading; we provide a fail fast payload validation to address this edge case.

With payload_validation_jmespath, you can provide an additional JMESPath expression to specify which part of the event body should be validated against previous idempotent invocations

=== "Payload validation"

```python hl_lines="12 20 28"
--8<-- "examples/idempotency/src/working_with_validation_payload.py"
```

=== "Sample event 1"

```json hl_lines="2 5"
--8<-- "examples/idempotency/src/working_with_validation_payload_payload1.json"
```

=== "Sample event 2"

```json hl_lines="2 5"
--8<-- "examples/idempotency/src/working_with_validation_payload_payload2.json"
```

In this example, the user_id and product_id keys are used as the payload to generate the idempotency key, as per event_key_jmespath parameter.

???+ note If we try to send the same request but with a different amount, we will raise IdempotencyValidationError.

Without payload validation, we would have returned the same result as we did for the initial request. Since we're also returning an amount in the response, this could be quite confusing for the client.

By using payload_validation_jmespath="amount", we prevent this potentially confusing behavior and instead raise an Exception.

Making idempotency key required

If you want to enforce that an idempotency key is required, you can set raise_on_no_idempotency_key to True.

This means that we will raise IdempotencyKeyError if the evaluation of event_key_jmespath is None.

???+ warning To prevent errors, transactions will not be treated as idempotent if raise_on_no_idempotency_key is set to False and the evaluation of event_key_jmespath is None. Therefore, no data will be fetched, stored, or deleted in the idempotency storage layer.

=== "Idempotency key required"

```python hl_lines="11"
--8<-- "examples/idempotency/src/working_with_idempotency_key_required.py"
```

=== "Success Event"

```json hl_lines="3 6"
--8<-- "examples/idempotency/src/working_with_idempotency_key_required_payload_success.json"
```

=== "Failure Event"

```json hl_lines="3 5"
--8<-- "examples/idempotency/src/working_with_idempotency_key_required_payload_error.json"
```

Customizing boto configuration

The boto_config and boto3_session parameters enable you to pass in a custom botocore config object{target="_blank"} or a custom boto3 session{target="_blank"} when constructing the persistence store.

=== "Custom session"

```python hl_lines="1 11 13"
--8<-- "examples/idempotency/src/working_with_custom_session.py"
```

=== "Custom config"

```python hl_lines="1 11 13"
--8<-- "examples/idempotency/src/working_with_custom_config.py"
```

=== "Sample Event"

```json
--8<-- "examples/idempotency/src/working_with_custom_config_payload.json"
```

Using a DynamoDB table with a composite primary key

When using a composite primary key table (hash+range key), use sort_key_attr parameter when initializing your persistence layer.

With this setting, we will save the idempotency key in the sort key instead of the primary key. By default, the primary key will now be set to idempotency#{LAMBDA_FUNCTION_NAME}.

You can optionally set a static value for the partition key using the static_pk_value parameter.

=== "Reusing a DynamoDB table that uses a composite primary key"

```python hl_lines="7"
--8<-- "examples/idempotency/src/working_with_composite_key.py"
```

=== "Sample Event"

```json
--8<-- "examples/idempotency/src/working_with_composite_key_payload.json"
```

The example function above would cause data to be stored in DynamoDB like this:

id	sort_key	expiration	status	data
idempotency#MyLambdaFunction	1e956ef7da78d0cb890be999aecc0c9e	1636549553	COMPLETED	{"user_id": 12391, "message": "success"}
idempotency#MyLambdaFunction	2b2cdb5f86361e97b4383087c1ffdf27	1636549571	COMPLETED	{"user_id": 527212, "message": "success"}
idempotency#MyLambdaFunction	f091d2527ad1c78f05d54cc3f363be80	1636549585	IN_PROGRESS

Bring your own persistent store

This utility provides an abstract base class (ABC), so that you can implement your choice of persistent storage layer.

You can create your own persistent store from scratch by inheriting the BasePersistenceLayer class, and implementing _get_record(), _put_record(), _update_record() and _delete_record().

_get_record() – Retrieves an item from the persistence store using an idempotency key and returns it as a DataRecord instance.
_put_record() – Adds a DataRecord to the persistence store if it doesn't already exist with that key. Raises an ItemAlreadyExists exception if a non-expired entry already exists.
_update_record() – Updates an item in the persistence store.
_delete_record() – Removes an item from the persistence store.

=== "Bring your own persistent store"

```python hl_lines="8 18 65 74 96 124"
--8<-- "examples/idempotency/src/bring_your_own_persistent_store.py"
```

???+ danger Pay attention to the documentation for each - you may need to perform additional checks inside these methods to ensure the idempotency guarantees remain intact.

For example, the `_put_record` method needs to raise an exception if a non-expired record already exists in the data store with a matching key.

Manipulating the Idempotent Response

You can set up a response_hook in the IdempotentConfig class to manipulate the returned data when an operation is idempotent. The hook function will be called with the current deserialized response object and the Idempotency record.

=== "Using an Idempotent Response Hook"

```python hl_lines="19 21 27 34"
--8<-- "examples/idempotency/src/working_with_response_hook.py"
```

=== "Sample event"

```json
--8<-- "examples/idempotency/src/working_with_response_hook_payload.json"
```

???+ info "Info: Using custom de-serialization?"

The response_hook is called after the custom de-serialization so the payload you process will be the de-serialized version.

Being a good citizen

When using response hooks to manipulate returned data from idempotent operations, it's important to follow best practices to avoid introducing complexity or issues. Keep these guidelines in mind:

Response hook works exclusively when operations are idempotent. The hook will not be called when an operation is not idempotent, or when the idempotent logic fails.
Catch and Handle Exceptions. Your response hook code should catch and handle any exceptions that may arise from your logic. Unhandled exceptions will cause the Lambda function to fail unexpectedly.
Keep Hook Logic Simple Response hooks should consist of minimal and straightforward logic for manipulating response data. Avoid complex conditional branching and aim for hooks that are easy to reason about.

Compatibility with other utilities

Batch

See Batch integration above.

Validation utility

The idempotency utility can be used with the validator decorator. Ensure that idempotency is the innermost decorator.

???+ warning If you use an envelope with the validator, the event received by the idempotency utility will be the unwrapped event - not the "raw" event Lambda was invoked with.

Make sure to account for this behavior, if you set the `event_key_jmespath`.

=== "Using Idempotency with JSONSchema Validation utility"

```python hl_lines="13"
--8<-- "examples/idempotency/src/integrate_idempotency_with_validator.py"
```

=== "Sample Event"

```json hl_lines="60"
--8<-- "examples/idempotency/src/integrate_idempotency_with_validator_payload.json"
```

???+ tip "Tip: JMESPath Powertools for AWS Lambda (Python) functions are also available" Built-in functions known in the validation utility like powertools_json, powertools_base64, powertools_base64_gzip are also available to use in this utility.

Tracer

The idempotency utility can be used with the tracer decorator. Ensure that idempotency is the innermost decorator.

First execution

During the first execution with a payload, Lambda performs a PutItem followed by an UpdateItem operation to persist the record in DynamoDB.

Subsequent executions

On subsequent executions with the same payload, Lambda optimistically tries to save the record in DynamoDB. If the record already exists, DynamoDB returns the item.

Explore how to handle conditional write errors in high-concurrency scenarios with DynamoDB in this blog post{target="_blank"}.

Testing your code

The idempotency utility provides several routes to test your code.

Disabling the idempotency utility

When testing your code, you may wish to disable the idempotency logic altogether and focus on testing your business logic. To do this, you can set the environment variable POWERTOOLS_IDEMPOTENCY_DISABLED with a truthy value. If you prefer setting this for specific tests, and are using Pytest, you can use monkeypatch{target="_blank" rel="nofollow"} fixture:

=== "test_disabling_idempotency_utility.py"

```python hl_lines="3 4 23 24"
--8<-- "examples/idempotency/tests/test_disabling_idempotency_utility.py"
```

=== "app_test_disabling_idempotency_utility.py"

```python hl_lines="10"
--8<-- "examples/idempotency/tests/app_test_disabling_idempotency_utility.py"
```

Testing with DynamoDB Local

To test with DynamoDB Local{target="_blank"}, you can replace the DynamoDB client used by the persistence layer with one you create inside your tests. This allows you to set the endpoint_url.

=== "test_with_dynamodb_local.py"

```python hl_lines="3-5 25 26"
--8<-- "examples/idempotency/tests/test_with_dynamodb_local.py"
```

=== "app_test_dynamodb_local.py"

```python hl_lines="10"
--8<-- "examples/idempotency/tests/app_test_dynamodb_local.py"
```

How do I mock all DynamoDB I/O operations

The idempotency utility lazily creates the dynamodb Table{target="_blank"} which it uses to access DynamoDB. This means it is possible to pass a mocked Table resource, or stub various methods.

=== "test_with_io_operations.py"

```python hl_lines="4 5 24 25 27"
--8<-- "examples/idempotency/tests/test_with_io_operations.py"
```

=== "app_test_io_operations.py"

```python hl_lines="10"
--8<-- "examples/idempotency/tests/app_test_io_operations.py"
```

Testing with Redis

To test locally, you can either utilize fakeredis-py for a simulated Redis environment or refer to the MockRedis class used in our tests to mock Redis operations.

=== "test_with_mock_redis.py"

```python hl_lines="2 3 29 31"
--8<-- "examples/idempotency/tests/test_with_mock_redis.py"
```

=== "mock_redis.py"

```python
--8<-- "examples/idempotency/tests/mock_redis.py"
```

If you want to set up a real Redis client for integration testing, you can reference the code provided below.

=== "test_with_real_redis.py"

```python hl_lines="3 4 29 38"
--8<-- "examples/idempotency/tests/test_with_real_redis.py"
```

=== "Makefile"

```bash
test-idempotency-redis: # (1)!
	docker run --name test-idempotency-redis -d -p 63005:6379 redis
	pytest test_with_real_redis.py;docker stop test-idempotency-redis;docker rm test-idempotency-redis
```

1. Use this script to setup a temp Redis docker and auto remove it upon completion

Extra resources

If you're interested in a deep dive on how Amazon uses idempotency when building our APIs, check out this article{target="_blank"}.

Files

idempotency.md

Latest commit

History

idempotency.md

File metadata and controls

Key features

Terminology

Getting started

IAM Permissions

Required resources

Idempotent decorator

Idempotent_function decorator

Output serialization

Batch integration

Choosing a payload subset for idempotency

Lambda timeouts

Handling exceptions

Persistence layers

DynamoDBPersistenceLayer

RedisPersistenceLayer

Idempotency request flow

Successful request

Successful request with cache enabled

Successful request with response_hook configured

Expired idempotency records

Concurrent identical in-flight requests

Lambda request timeout

Optional idempotency key

Race condition with Redis

Redis as persistent storage layer provider

Redis resources

VPC Access

Configuring Redis persistence layer

Custom advanced settings

Advanced

Customizing the default behavior

Handling concurrent executions with the same payload

Using in-memory cache

Expiring idempotency records

Payload validation

Making idempotency key required

Customizing boto configuration

Using a DynamoDB table with a composite primary key

Bring your own persistent store

Manipulating the Idempotent Response

Being a good citizen

Compatibility with other utilities

Batch

Validation utility

Tracer

First execution

Subsequent executions

Testing your code

Disabling the idempotency utility

Testing with DynamoDB Local

How do I mock all DynamoDB I/O operations

Testing with Redis

Extra resources