Skip to content

Commit 20d55c3

Browse files
seshubawsseshubaws
authored
Merge branch 'develop' into nested_event_sources
2 parents d406f43 + b149b15 commit 20d55c3

File tree

10 files changed

+316
-54
lines changed

10 files changed

+316
-54
lines changed

‎CHANGELOG.md

+13-7
Original file line numberDiff line numberDiff line change
@@ -10,23 +10,29 @@
1010

1111
## Maintenance
1212

13-
* **deps:** bump squidfunk/mkdocs-material from `065f3af` to `6b124e1` in /docs ([#4055](https://github.com/aws-powertools/powertools-lambda-python/issues/4055))
13+
* **ci:** prevent deprecated custom runner from being used ([#4061](https://github.com/aws-powertools/powertools-lambda-python/issues/4061))
1414
* **deps:** bump datadog-lambda from 5.91.0 to 5.92.0 ([#4038](https://github.com/aws-powertools/powertools-lambda-python/issues/4038))
15-
* **deps:** bump squidfunk/mkdocs-material from `3307665` to `065f3af` in /docs ([#4052](https://github.com/aws-powertools/powertools-lambda-python/issues/4052))
1615
* **deps:** bump the layer-balancer group in /layer/scripts/layer-balancer with 3 updates ([#4042](https://github.com/aws-powertools/powertools-lambda-python/issues/4042))
17-
* **deps-dev:** bump cdklabs-generative-ai-cdk-constructs from 0.1.105 to 0.1.106 ([#4048](https://github.com/aws-powertools/powertools-lambda-python/issues/4048))
16+
* **deps:** bump the layer-balancer group in /layer/scripts/layer-balancer with 1 update ([#4066](https://github.com/aws-powertools/powertools-lambda-python/issues/4066))
17+
* **deps:** bump squidfunk/mkdocs-material from `065f3af` to `6b124e1` in /docs ([#4055](https://github.com/aws-powertools/powertools-lambda-python/issues/4055))
18+
* **deps:** bump pydantic from 1.10.14 to 1.10.15 ([#4064](https://github.com/aws-powertools/powertools-lambda-python/issues/4064))
19+
* **deps:** bump squidfunk/mkdocs-material from `3307665` to `065f3af` in /docs ([#4052](https://github.com/aws-powertools/powertools-lambda-python/issues/4052))
1820
* **deps-dev:** bump the boto-typing group with 1 update ([#4047](https://github.com/aws-powertools/powertools-lambda-python/issues/4047))
1921
* **deps-dev:** bump mkdocs-material from 9.5.15 to 9.5.16 ([#4050](https://github.com/aws-powertools/powertools-lambda-python/issues/4050))
20-
* **deps-dev:** bump aws-cdk-aws-lambda-python-alpha from 2.133.0a0 to 2.134.0a0 ([#4039](https://github.com/aws-powertools/powertools-lambda-python/issues/4039))
21-
* **deps-dev:** bump sentry-sdk from 1.43.0 to 1.44.0 ([#4040](https://github.com/aws-powertools/powertools-lambda-python/issues/4040))
2222
* **deps-dev:** bump ruff from 0.3.4 to 0.3.5 ([#4049](https://github.com/aws-powertools/powertools-lambda-python/issues/4049))
23+
* **deps-dev:** bump cdklabs-generative-ai-cdk-constructs from 0.1.105 to 0.1.106 ([#4048](https://github.com/aws-powertools/powertools-lambda-python/issues/4048))
2324
* **deps-dev:** bump mkdocs-material from 9.5.16 to 9.5.17 ([#4056](https://github.com/aws-powertools/powertools-lambda-python/issues/4056))
25+
* **deps-dev:** bump aws-cdk-lib from 2.134.0 to 2.135.0 ([#4057](https://github.com/aws-powertools/powertools-lambda-python/issues/4057))
26+
* **deps-dev:** bump aws-cdk-aws-lambda-python-alpha from 2.133.0a0 to 2.134.0a0 ([#4039](https://github.com/aws-powertools/powertools-lambda-python/issues/4039))
27+
* **deps-dev:** bump sentry-sdk from 1.43.0 to 1.44.0 ([#4040](https://github.com/aws-powertools/powertools-lambda-python/issues/4040))
28+
* **deps-dev:** bump aws-cdk from 2.134.0 to 2.135.0 ([#4058](https://github.com/aws-powertools/powertools-lambda-python/issues/4058))
29+
* **deps-dev:** bump the boto-typing group with 2 updates ([#4062](https://github.com/aws-powertools/powertools-lambda-python/issues/4062))
2430
* **deps-dev:** bump aws-cdk-lib from 2.133.0 to 2.134.0 ([#4031](https://github.com/aws-powertools/powertools-lambda-python/issues/4031))
2531
* **deps-dev:** bump cdklabs-generative-ai-cdk-constructs from 0.1.104 to 0.1.105 ([#4030](https://github.com/aws-powertools/powertools-lambda-python/issues/4030))
2632
* **deps-dev:** bump aws-cdk from 2.133.0 to 2.134.0 ([#4032](https://github.com/aws-powertools/powertools-lambda-python/issues/4032))
2733
* **deps-dev:** bump the boto-typing group with 1 update ([#4029](https://github.com/aws-powertools/powertools-lambda-python/issues/4029))
28-
* **deps-dev:** bump aws-cdk-lib from 2.134.0 to 2.135.0 ([#4057](https://github.com/aws-powertools/powertools-lambda-python/issues/4057))
29-
* **deps-dev:** bump aws-cdk from 2.134.0 to 2.135.0 ([#4058](https://github.com/aws-powertools/powertools-lambda-python/issues/4058))
34+
* **deps-dev:** bump aws-cdk-aws-lambda-python-alpha from 2.134.0a0 to 2.135.0a0 ([#4063](https://github.com/aws-powertools/powertools-lambda-python/issues/4063))
35+
* **deps-dev:** bump sentry-sdk from 1.44.0 to 1.44.1 ([#4065](https://github.com/aws-powertools/powertools-lambda-python/issues/4065))
3036

3137

3238
<a name="v2.36.0"></a>

‎aws_lambda_powertools/utilities/idempotency/__init__.py

+4
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,9 @@
22
Utility for adding idempotency to lambda functions
33
"""
44

5+
from aws_lambda_powertools.utilities.idempotency.hook import (
6+
IdempotentHookFunction,
7+
)
58
from aws_lambda_powertools.utilities.idempotency.persistence.base import (
69
BasePersistenceLayer,
710
)
@@ -17,4 +20,5 @@
1720
"idempotent",
1821
"idempotent_function",
1922
"IdempotencyConfig",
23+
"IdempotentHookFunction",
2024
)

‎aws_lambda_powertools/utilities/idempotency/base.py

+12-2
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,9 @@
33
from copy import deepcopy
44
from typing import Any, Callable, Dict, Optional, Tuple
55

6-
from aws_lambda_powertools.utilities.idempotency.config import IdempotencyConfig
6+
from aws_lambda_powertools.utilities.idempotency.config import (
7+
IdempotencyConfig,
8+
)
79
from aws_lambda_powertools.utilities.idempotency.exceptions import (
810
IdempotencyAlreadyInProgressError,
911
IdempotencyInconsistentStateError,
@@ -227,7 +229,15 @@ def _handle_for_status(self, data_record: DataRecord) -> Optional[Any]:
227229
)
228230
response_dict: Optional[dict] = data_record.response_json_as_dict()
229231
if response_dict is not None:
230-
return self.output_serializer.from_dict(response_dict)
232+
serialized_response = self.output_serializer.from_dict(response_dict)
233+
if self.config.response_hook is not None:
234+
logger.debug("Response hook configured, invoking function")
235+
return self.config.response_hook(
236+
serialized_response,
237+
data_record,
238+
)
239+
return serialized_response
240+
231241
return None
232242

233243
def _get_function_response(self):

‎aws_lambda_powertools/utilities/idempotency/config.py

+5
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
from typing import Dict, Optional
22

3+
from aws_lambda_powertools.utilities.idempotency import IdempotentHookFunction
34
from aws_lambda_powertools.utilities.typing import LambdaContext
45

56

@@ -15,6 +16,7 @@ def __init__(
1516
local_cache_max_items: int = 256,
1617
hash_function: str = "md5",
1718
lambda_context: Optional[LambdaContext] = None,
19+
response_hook: Optional[IdempotentHookFunction] = None,
1820
):
1921
"""
2022
Initialize the base persistence layer
@@ -37,6 +39,8 @@ def __init__(
3739
Function to use for calculating hashes, by default md5.
3840
lambda_context: LambdaContext, optional
3941
Lambda Context containing information about the invocation, function and execution environment.
42+
response_hook: IdempotentHookFunction, optional
43+
Hook function to be called when an idempotent response is returned from the idempotent store.
4044
"""
4145
self.event_key_jmespath = event_key_jmespath
4246
self.payload_validation_jmespath = payload_validation_jmespath
@@ -47,6 +51,7 @@ def __init__(
4751
self.local_cache_max_items = local_cache_max_items
4852
self.hash_function = hash_function
4953
self.lambda_context: Optional[LambdaContext] = lambda_context
54+
self.response_hook: Optional[IdempotentHookFunction] = response_hook
5055

5156
def register_lambda_context(self, lambda_context: LambdaContext):
5257
"""Captures the Lambda context, to calculate the remaining time before the invocation times out"""

‎aws_lambda_powertools/utilities/idempotency/hook.py

+13
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
from typing import Any
2+
3+
from aws_lambda_powertools.shared.types import Protocol
4+
from aws_lambda_powertools.utilities.idempotency.persistence.datarecord import DataRecord
5+
6+
7+
class IdempotentHookFunction(Protocol):
8+
"""
9+
The IdempotentHookFunction.
10+
This class defines the calling signature for IdempotentHookFunction callbacks.
11+
"""
12+
13+
def __call__(self, response: Any, idempotent_data: DataRecord) -> Any: ...

‎docs/utilities/idempotency.md

+76-11
Original file line numberDiff line numberDiff line change
@@ -73,8 +73,8 @@ We currently support Amazon DynamoDB and Redis as a storage layer. The following
7373
If you're not [changing the default configuration for the DynamoDB persistence layer](#dynamodbpersistencelayer), this is the expected default configuration:
7474

7575
| Configuration | Value | Notes |
76-
| ------------------ | ------------ | ----------------------------------------------------------------------------------- |
77-
| Partition key | `id` |
76+
| ------------------ | ------------ |-------------------------------------------------------------------------------------|
77+
| Partition key | `id` | |
7878
| TTL attribute name | `expiration` | This can only be configured after your table is created if you're using AWS Console |
7979

8080
???+ tip "Tip: You can share a single state table for all functions"
@@ -454,6 +454,40 @@ sequenceDiagram
454454
<i>Idempotent successful request cached</i>
455455
</center>
456456

457+
#### Successful request with response_hook configured
458+
459+
<center>
460+
```mermaid
461+
sequenceDiagram
462+
participant Client
463+
participant Lambda
464+
participant Response hook
465+
participant Persistence Layer
466+
alt initial request
467+
Client->>Lambda: Invoke (event)
468+
Lambda->>Persistence Layer: Get or set idempotency_key=hash(payload)
469+
activate Persistence Layer
470+
Note over Lambda,Persistence Layer: Set record status to INPROGRESS. <br> Prevents concurrent invocations <br> with the same payload
471+
Lambda-->>Lambda: Call your function
472+
Lambda->>Persistence Layer: Update record with result
473+
deactivate Persistence Layer
474+
Persistence Layer-->>Persistence Layer: Update record
475+
Note over Lambda,Persistence Layer: Set record status to COMPLETE. <br> New invocations with the same payload <br> now return the same result
476+
Lambda-->>Client: Response sent to client
477+
else retried request
478+
Client->>Lambda: Invoke (event)
479+
Lambda->>Persistence Layer: Get or set idempotency_key=hash(payload)
480+
activate Persistence Layer
481+
Persistence Layer-->>Response hook: Already exists in persistence layer.
482+
deactivate Persistence Layer
483+
Note over Response hook,Persistence Layer: Record status is COMPLETE and not expired
484+
Response hook->>Lambda: Response hook invoked
485+
Lambda-->>Client: Manipulated idempotent response sent to client
486+
end
487+
```
488+
<i>Successful idempotent request with a response hook</i>
489+
</center>
490+
457491
#### Expired idempotency records
458492

459493
<center>
@@ -699,15 +733,16 @@ For advanced configurations, such as setting up SSL certificates or customizing
699733

700734
Idempotent decorator can be further configured with **`IdempotencyConfig`** as seen in the previous example. These are the available options for further configuration
701735

702-
| Parameter | Default | Description |
703-
| ------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
704-
| **event_key_jmespath** | `""` | JMESPath expression to extract the idempotency key from the event record using [built-in functions](./jmespath_functions.md#built-in-jmespath-functions){target="_blank"} |
705-
| **payload_validation_jmespath** | `""` | JMESPath expression to validate whether certain parameters have changed in the event while the event payload |
706-
| **raise_on_no_idempotency_key** | `False` | Raise exception if no idempotency key was found in the request |
707-
| **expires_after_seconds** | 3600 | The number of seconds to wait before a record is expired |
708-
| **use_local_cache** | `False` | Whether to locally cache idempotency results |
709-
| **local_cache_max_items** | 256 | Max number of items to store in local cache |
710-
| **hash_function** | `md5` | Function to use for calculating hashes, as provided by [hashlib](https://docs.python.org/3/library/hashlib.html){target="_blank" rel="nofollow"} in the standard library. |
736+
| Parameter | Default | Description |
737+
|---------------------------------|---------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
738+
| **event_key_jmespath** | `""` | JMESPath expression to extract the idempotency key from the event record using [built-in functions](./jmespath_functions.md#built-in-jmespath-functions){target="_blank"} |
739+
| **payload_validation_jmespath** | `""` | JMESPath expression to validate whether certain parameters have changed in the event while the event payload |
740+
| **raise_on_no_idempotency_key** | `False` | Raise exception if no idempotency key was found in the request |
741+
| **expires_after_seconds** | 3600 | The number of seconds to wait before a record is expired |
742+
| **use_local_cache** | `False` | Whether to locally cache idempotency results |
743+
| **local_cache_max_items** | 256 | Max number of items to store in local cache |
744+
| **hash_function** | `md5` | Function to use for calculating hashes, as provided by [hashlib](https://docs.python.org/3/library/hashlib.html){target="_blank" rel="nofollow"} in the standard library. |
745+
| **response_hook** | `None` | Function to use for processing the stored Idempotent response. This function hook is called when an existing idempotent response is found. See [Manipulating The Idempotent Response](idempotency.md#manipulating-the-idempotent-response) |
711746

712747
### Handling concurrent executions with the same payload
713748

@@ -909,6 +944,36 @@ You can create your own persistent store from scratch by inheriting the `BasePer
909944

910945
For example, the `_put_record` method needs to raise an exception if a non-expired record already exists in the data store with a matching key.
911946

947+
### Manipulating the Idempotent Response
948+
949+
You can set up a `response_hook` in the `IdempotentConfig` class to manipulate the returned data when an operation is idempotent. The hook function will be called with the current deserialized response object and the Idempotency record.
950+
951+
=== "Using an Idempotent Response Hook"
952+
953+
```python hl_lines="18 20 23 32"
954+
--8<-- "examples/idempotency/src/working_with_response_hook.py"
955+
```
956+
957+
=== "Sample event"
958+
959+
```json
960+
--8<-- "examples/idempotency/src/working_with_response_hook_payload.json"
961+
```
962+
963+
???+ info "Info: Using custom de-serialization?"
964+
965+
The response_hook is called after the custom de-serialization so the payload you process will be the de-serialized version.
966+
967+
#### Being a good citizen
968+
969+
When using response hooks to manipulate returned data from idempotent operations, it's important to follow best practices to avoid introducing complexity or issues. Keep these guidelines in mind:
970+
971+
1. **Response hook works exclusively when operations are idempotent.** The hook will not be called when an operation is not idempotent, or when the idempotent logic fails.
972+
973+
2. **Catch and Handle Exceptions.** Your response hook code should catch and handle any exceptions that may arise from your logic. Unhandled exceptions will cause the Lambda function to fail unexpectedly.
974+
975+
3. **Keep Hook Logic Simple** Response hooks should consist of minimal and straightforward logic for manipulating response data. Avoid complex conditional branching and aim for hooks that are easy to reason about.
976+
912977
## Compatibility with other utilities
913978

914979
### Batch

‎docs/we_made_this.md

+8
Original file line numberDiff line numberDiff line change
@@ -114,6 +114,14 @@ This article will guide you through personalizing observability by integrating C
114114
115115
[Creating a serverless API using Powertools for AWS Lambda and CDK](https://www.ranthebuilder.cloud/post/boost-app-engagement-with-aws-cloudwatch-metrics-powertools-for-aws){target="_blank" rel="nofollow"}
116116

117+
### Streaming data with AWS Lambda & Powertools for AWS Lambda
118+
119+
This article will walk you through using Powertools for AWS Lambda to optimize your Lambda function when streaming large files from S3.
120+
121+
> **Author: [Tom Reid](https://www.linkedin.com/in/tom-reid-5a2a3a/){target="_blank" rel="nofollow"}** :material-linkedin:
122+
123+
[Streaming data with AWS Lambda & Powertools for AWS Lambda](https://towardsdev.com/streaming-data-with-aws-lambda-5f0e81f854cd){target="_blank" rel="nofollow"}
124+
117125
## Videos
118126

119127
#### Building a resilient input handling with Parser

0 commit comments

Comments
 (0)