Skip to content

Commit f4ea4f8

Browse files
heitorlessaleandrodamascena
heitorlessa
and
leandrodamascena
authored
docs(idempotency): improve navigation, wording, and new section on guarantees (#4613)
* docs(idempotency): cleanup redis usage and link with setup/infra * docs(idempotency): cleanup idempotent decorator; inline admonitions Signed-off-by: heitorlessa <[email protected]> * docs(idempotency): cleanup idempotent_decorator section Signed-off-by: heitorlessa <[email protected]> * docs(idempotency): cleanup serialization, fields subset, move batch to new common use cases section Signed-off-by: heitorlessa <[email protected]> * docs: cleanup handling exceptions Signed-off-by: heitorlessa <[email protected]> * docs: move caching to getting started * docs: use env var for DDB table, no hardcode * docs: moved expiration window to getting started; updated example to set to 24h * docs: include IdempotencyValidationError in example * Fixing errors on Redis examples * docs(config): add social links * docs(idempotency): cleanup intro and key features * docs(idempotency): cleanup getting started ddb vs redis * docs(idempotency): break iam permissions into table; IAM permission to clipboard * docs(idempotency): cleanup dynamodb required resource; break subsections and update nav * docs(idempotency): make terminologies crispier Signed-off-by: heitorlessa <[email protected]> * docs(idempotency): line editing before decorators Signed-off-by: heitorlessa <[email protected]> * docs(idempotency): cleanup timeout section Signed-off-by: heitorlessa <[email protected]> * docs(idempotency): use cards for required resources Signed-off-by: heitorlessa <[email protected]> * docs(idempotency): note to skip timeout section when using handler decorator Signed-off-by: heitorlessa <[email protected]> * docs: remove tabbed content for single timeout snippet Signed-off-by: heitorlessa <[email protected]> * docs: cleanup persistence layers attrs, snippet titles etc Signed-off-by: heitorlessa <[email protected]> * docs: typo in batch integration Signed-off-by: heitorlessa <[email protected]> * docs: rename batch integration to actual use case name Signed-off-by: heitorlessa <[email protected]> * docs(idempotency): cleanup default behavior section Signed-off-by: heitorlessa <[email protected]> * docs: move bold to draw attention to whole event as idempotency key Signed-off-by: heitorlessa <[email protected]> * docs: lead with parameter name over config name * docs: moved composite key under DDB section * docs: cut unnecessary anchor name * docs: fix broken links after sections renaming * Making mypy happy * docs: fix conflicts out of order * docs(leandro's feedback): add caching in key features * Apply suggestions from code review Co-authored-by: Leandro Damascena <[email protected]> Signed-off-by: Heitor Lessa <[email protected]> * docs(leandro's feedback): time window placement * docs(leandro's feedback): key features success vs failure ambiguity * docs(leandro's feedback): Redis anchor name * docs(leandro's feedback): remove ambiguity on Redis VPC connectivity * Update docs/utilities/idempotency.md Co-authored-by: Leandro Damascena <[email protected]> Signed-off-by: Heitor Lessa <[email protected]> * Update docs/utilities/idempotency.md Co-authored-by: Leandro Damascena <[email protected]> Signed-off-by: Heitor Lessa <[email protected]> * docs: move primary key for both persistence storages plus additional ctx --------- Signed-off-by: heitorlessa <[email protected]> Signed-off-by: Heitor Lessa <[email protected]> Co-authored-by: Leandro Damascena <[email protected]>
1 parent 39c0314 commit f4ea4f8

33 files changed

+524
-444
lines changed

Diff for: ‎docs/utilities/idempotency.md

+330-339
Large diffs are not rendered by default.

Diff for: ‎examples/idempotency/src/customize_persistence_layer.py

+4-1
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,14 @@
1+
import os
2+
13
from aws_lambda_powertools.utilities.idempotency import (
24
DynamoDBPersistenceLayer,
35
idempotent,
46
)
57
from aws_lambda_powertools.utilities.typing import LambdaContext
68

9+
table = os.getenv("IDEMPOTENCY_TABLE", "")
710
persistence_layer = DynamoDBPersistenceLayer(
8-
table_name="IdempotencyTable",
11+
table_name=table,
912
key_attr="idempotency_key",
1013
expiry_attr="expires_at",
1114
in_progress_expiry_attr="in_progress_expires_at",

Diff for: ‎examples/idempotency/src/customize_persistence_layer_redis.py

+4-1
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
import os
2+
13
from aws_lambda_powertools.utilities.idempotency import (
24
idempotent,
35
)
@@ -6,8 +8,9 @@
68
)
79
from aws_lambda_powertools.utilities.typing import LambdaContext
810

11+
redis_endpoint = os.getenv("REDIS_CLUSTER_ENDPOINT", "localhost")
912
persistence_layer = RedisCachePersistenceLayer(
10-
host="localhost",
13+
host=redis_endpoint,
1114
port=6379,
1215
in_progress_expiry_attr="in_progress_expiration",
1316
status_attr="status",

Diff for: ‎examples/idempotency/src/getting_started_with_idempotency.py

+4-3
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import os
12
from dataclasses import dataclass, field
23
from uuid import uuid4
34

@@ -7,7 +8,8 @@
78
)
89
from aws_lambda_powertools.utilities.typing import LambdaContext
910

10-
persistence_layer = DynamoDBPersistenceLayer(table_name="IdempotencyTable")
11+
table = os.getenv("IDEMPOTENCY_TABLE", "")
12+
persistence_layer = DynamoDBPersistenceLayer(table_name=table)
1113

1214

1315
@dataclass
@@ -17,8 +19,7 @@ class Payment:
1719
payment_id: str = field(default_factory=lambda: f"{uuid4()}")
1820

1921

20-
class PaymentError(Exception):
21-
...
22+
class PaymentError(Exception): ...
2223

2324

2425
@idempotent(persistence_store=persistence_layer)

Diff for: ‎examples/idempotency/src/getting_started_with_idempotency_redis_client.py

+4-3
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import os
12
from dataclasses import dataclass, field
23
from uuid import uuid4
34

@@ -11,8 +12,9 @@
1112
)
1213
from aws_lambda_powertools.utilities.typing import LambdaContext
1314

15+
redis_endpoint = os.getenv("REDIS_CLUSTER_ENDPOINT", "localhost")
1416
client = Redis(
15-
host="localhost",
17+
host=redis_endpoint,
1618
port=6379,
1719
socket_connect_timeout=5,
1820
socket_timeout=5,
@@ -29,8 +31,7 @@ class Payment:
2931
payment_id: str = field(default_factory=lambda: f"{uuid4()}")
3032

3133

32-
class PaymentError(Exception):
33-
...
34+
class PaymentError(Exception): ...
3435

3536

3637
@idempotent(persistence_store=persistence_layer)

Diff for: ‎examples/idempotency/src/getting_started_with_idempotency_redis_config.py

+4-3
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import os
12
from dataclasses import dataclass, field
23
from uuid import uuid4
34

@@ -9,7 +10,8 @@
910
)
1011
from aws_lambda_powertools.utilities.typing import LambdaContext
1112

12-
persistence_layer = RedisCachePersistenceLayer(host="localhost", port=6379)
13+
redis_endpoint = os.getenv("REDIS_CLUSTER_ENDPOINT", "localhost")
14+
persistence_layer = RedisCachePersistenceLayer(host=redis_endpoint, port=6379)
1315

1416

1517
@dataclass
@@ -19,8 +21,7 @@ class Payment:
1921
payment_id: str = field(default_factory=lambda: f"{uuid4()}")
2022

2123

22-
class PaymentError(Exception):
23-
...
24+
class PaymentError(Exception): ...
2425

2526

2627
@idempotent(persistence_store=persistence_layer)

Diff for: ‎examples/idempotency/src/integrate_idempotency_with_batch_processor.py

+14-18
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
1-
from aws_lambda_powertools import Logger
2-
from aws_lambda_powertools.utilities.batch import BatchProcessor, EventType
1+
import os
2+
from typing import Any, Dict
3+
4+
from aws_lambda_powertools.utilities.batch import BatchProcessor, EventType, process_partial_response
35
from aws_lambda_powertools.utilities.data_classes.sqs_event import SQSRecord
46
from aws_lambda_powertools.utilities.idempotency import (
57
DynamoDBPersistenceLayer,
@@ -8,30 +10,24 @@
810
)
911
from aws_lambda_powertools.utilities.typing import LambdaContext
1012

11-
logger = Logger()
1213
processor = BatchProcessor(event_type=EventType.SQS)
1314

14-
dynamodb = DynamoDBPersistenceLayer(table_name="IdempotencyTable")
15-
config = IdempotencyConfig(
16-
event_key_jmespath="messageId", # see Choosing a payload subset section
17-
)
15+
table = os.getenv("IDEMPOTENCY_TABLE", "")
16+
dynamodb = DynamoDBPersistenceLayer(table_name=table)
17+
config = IdempotencyConfig(event_key_jmespath="messageId")
1818

1919

2020
@idempotent_function(data_keyword_argument="record", config=config, persistence_store=dynamodb)
2121
def record_handler(record: SQSRecord):
2222
return {"message": record.body}
2323

2424

25-
def lambda_handler(event: SQSRecord, context: LambdaContext):
25+
def lambda_handler(event: Dict[str, Any], context: LambdaContext):
2626
config.register_lambda_context(context) # see Lambda timeouts section
2727

28-
# with Lambda context registered for Idempotency
29-
# we can now kick in the Bach processing logic
30-
batch = event["Records"]
31-
with processor(records=batch, handler=record_handler):
32-
# in case you want to access each record processed by your record_handler
33-
# otherwise ignore the result variable assignment
34-
processed_messages = processor.process()
35-
logger.info(processed_messages)
36-
37-
return processor.response()
28+
return process_partial_response(
29+
event=event,
30+
context=context,
31+
processor=processor,
32+
record_handler=record_handler,
33+
)

Diff for: ‎examples/idempotency/src/integrate_idempotency_with_validator.py

+4-1
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
import os
2+
13
from aws_lambda_powertools.utilities.idempotency import (
24
DynamoDBPersistenceLayer,
35
IdempotencyConfig,
@@ -6,8 +8,9 @@
68
from aws_lambda_powertools.utilities.typing import LambdaContext
79
from aws_lambda_powertools.utilities.validation import envelopes, validator
810

11+
table = os.getenv("IDEMPOTENCY_TABLE", "")
912
config = IdempotencyConfig(event_key_jmespath='["message", "username"]')
10-
persistence_layer = DynamoDBPersistenceLayer(table_name="IdempotencyTable")
13+
persistence_layer = DynamoDBPersistenceLayer(table_name=table)
1114

1215

1316
@validator(envelope=envelopes.API_GATEWAY_HTTP)

Diff for: ‎examples/idempotency/src/using_redis_client_with_aws_secrets.py

+5-3
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
from __future__ import annotations
2+
13
from typing import Any
24

35
from redis import Redis
@@ -8,11 +10,11 @@
810
RedisCachePersistenceLayer,
911
)
1012

11-
redis_values: Any = parameters.get_secret("redis_info", transform="json") # (1)!
13+
redis_values: dict[str, Any] = parameters.get_secret("redis_info", transform="json") # (1)!
1214

1315
redis_client = Redis(
14-
host=redis_values.get("REDIS_HOST"),
15-
port=redis_values.get("REDIS_PORT"),
16+
host=redis_values.get("REDIS_HOST", "localhost"),
17+
port=redis_values.get("REDIS_PORT", 6379),
1618
password=redis_values.get("REDIS_PASSWORD"),
1719
decode_responses=True,
1820
socket_timeout=10.0,

Diff for: ‎examples/idempotency/src/using_redis_client_with_local_certs.py

+5-3
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
from __future__ import annotations
2+
13
from typing import Any
24

35
from redis import Redis
@@ -9,12 +11,12 @@
911
RedisCachePersistenceLayer,
1012
)
1113

12-
redis_values: Any = parameters.get_secret("redis_info", transform="json") # (1)!
14+
redis_values: dict[str, Any] = parameters.get_secret("redis_info", transform="json") # (1)!
1315

1416

1517
redis_client = Redis(
16-
host=redis_values.get("REDIS_HOST"),
17-
port=redis_values.get("REDIS_PORT"),
18+
host=redis_values.get("REDIS_HOST", "localhost"),
19+
port=redis_values.get("REDIS_PORT", 6379),
1820
password=redis_values.get("REDIS_PASSWORD"),
1921
decode_responses=True,
2022
socket_timeout=10.0,

Diff for: ‎examples/idempotency/src/working_with_composite_key.py

+4-1
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,13 @@
1+
import os
2+
13
from aws_lambda_powertools.utilities.idempotency import (
24
DynamoDBPersistenceLayer,
35
idempotent,
46
)
57
from aws_lambda_powertools.utilities.typing import LambdaContext
68

7-
persistence_layer = DynamoDBPersistenceLayer(table_name="IdempotencyTable", sort_key_attr="sort_key")
9+
table = os.getenv("IDEMPOTENCY_TABLE", "")
10+
persistence_layer = DynamoDBPersistenceLayer(table_name=table, sort_key_attr="sort_key")
811

912

1013
@idempotent(persistence_store=persistence_layer)

Diff for: ‎examples/idempotency/src/working_with_custom_config.py

+4-1
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
import os
2+
13
from botocore.config import Config
24

35
from aws_lambda_powertools.utilities.idempotency import (
@@ -10,7 +12,8 @@
1012
# See: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html#botocore-config
1113
boto_config = Config()
1214

13-
persistence_layer = DynamoDBPersistenceLayer(table_name="IdempotencyTable", boto_config=boto_config)
15+
table = os.getenv("IDEMPOTENCY_TABLE", "")
16+
persistence_layer = DynamoDBPersistenceLayer(table_name=table, boto_config=boto_config)
1417

1518
config = IdempotencyConfig(event_key_jmespath="body")
1619

Diff for: ‎examples/idempotency/src/working_with_custom_session.py

+4-1
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
import os
2+
13
import boto3
24

35
from aws_lambda_powertools.utilities.idempotency import (
@@ -10,7 +12,8 @@
1012
# See: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html#module-boto3.session
1113
boto3_session = boto3.session.Session()
1214

13-
persistence_layer = DynamoDBPersistenceLayer(table_name="IdempotencyTable", boto3_session=boto3_session)
15+
table = os.getenv("IDEMPOTENCY_TABLE", "")
16+
persistence_layer = DynamoDBPersistenceLayer(table_name=table, boto3_session=boto3_session)
1417

1518
config = IdempotencyConfig(event_key_jmespath="body")
1619

Diff for: ‎examples/idempotency/src/working_with_dataclass_deduced_output_serializer.py

+3-1
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import os
12
from dataclasses import dataclass
23

34
from aws_lambda_powertools.utilities.idempotency import (
@@ -8,7 +9,8 @@
89
from aws_lambda_powertools.utilities.idempotency.serialization.dataclass import DataclassSerializer
910
from aws_lambda_powertools.utilities.typing import LambdaContext
1011

11-
dynamodb = DynamoDBPersistenceLayer(table_name="IdempotencyTable")
12+
table = os.getenv("IDEMPOTENCY_TABLE", "")
13+
dynamodb = DynamoDBPersistenceLayer(table_name=table)
1214
config = IdempotencyConfig(event_key_jmespath="order_id") # see Choosing a payload subset section
1315

1416

Diff for: ‎examples/idempotency/src/working_with_dataclass_explicitly_output_serializer.py

+3-1
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import os
12
from dataclasses import dataclass
23

34
from aws_lambda_powertools.utilities.idempotency import (
@@ -8,7 +9,8 @@
89
from aws_lambda_powertools.utilities.idempotency.serialization.dataclass import DataclassSerializer
910
from aws_lambda_powertools.utilities.typing import LambdaContext
1011

11-
dynamodb = DynamoDBPersistenceLayer(table_name="IdempotencyTable")
12+
table = os.getenv("IDEMPOTENCY_TABLE", "")
13+
dynamodb = DynamoDBPersistenceLayer(table_name=table)
1214
config = IdempotencyConfig(event_key_jmespath="order_id") # see Choosing a payload subset section
1315

1416

Diff for: ‎examples/idempotency/src/working_with_exceptions.py

+19-18
Original file line numberDiff line numberDiff line change
@@ -1,37 +1,38 @@
1+
import os
2+
13
import requests
24

35
from aws_lambda_powertools.utilities.idempotency import (
46
DynamoDBPersistenceLayer,
57
IdempotencyConfig,
68
idempotent_function,
79
)
10+
from aws_lambda_powertools.utilities.idempotency.exceptions import IdempotencyPersistenceLayerError
811
from aws_lambda_powertools.utilities.typing import LambdaContext
912

10-
persistence_layer = DynamoDBPersistenceLayer(table_name="IdempotencyTable")
13+
table = os.getenv("IDEMPOTENCY_TABLE", "")
14+
persistence_layer = DynamoDBPersistenceLayer(table_name=table)
1115

1216
config = IdempotencyConfig()
1317

1418

15-
def lambda_handler(event: dict, context: LambdaContext):
16-
# If an exception is raised here, no idempotent record will ever get created as the
17-
# idempotent function does not get called
18-
try:
19-
endpoint = "https://jsonplaceholder.typicode.com/comments/" # change this endpoint to force an exception
20-
requests.get(endpoint)
21-
except Exception as exc:
22-
return str(exc)
23-
24-
call_external_service(data={"user": "user1", "id": 5})
25-
26-
# This exception will not cause the idempotent record to be deleted, since it
27-
# happens after the decorated function has been successfully called
28-
raise Exception
29-
30-
3119
@idempotent_function(data_keyword_argument="data", config=config, persistence_store=persistence_layer)
3220
def call_external_service(data: dict):
21+
# Any exception raised will lead to idempotency record to be deleted
3322
result: requests.Response = requests.post(
3423
"https://jsonplaceholder.typicode.com/comments/",
35-
json={"user": data["user"], "transaction_id": data["id"]},
24+
json=data,
3625
)
3726
return result.json()
27+
28+
29+
def lambda_handler(event: dict, context: LambdaContext):
30+
try:
31+
call_external_service(data=event)
32+
except IdempotencyPersistenceLayerError as e:
33+
# No idempotency, but you can decide to error differently.
34+
raise RuntimeError(f"Oops, can't talk to persistence layer. Permissions? error: {e}")
35+
36+
# This exception will not impact the idempotency of 'call_external_service'
37+
# because it happens in isolation, or outside their scope.
38+
raise SyntaxError("Oops, this shouldn't be here.")

Diff for: ‎examples/idempotency/src/working_with_idempotency_key_required.py

+4-1
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,14 @@
1+
import os
2+
13
from aws_lambda_powertools.utilities.idempotency import (
24
DynamoDBPersistenceLayer,
35
IdempotencyConfig,
46
idempotent,
57
)
68
from aws_lambda_powertools.utilities.typing import LambdaContext
79

8-
persistence_layer = DynamoDBPersistenceLayer(table_name="IdempotencyTable")
10+
table = os.getenv("IDEMPOTENCY_TABLE", "")
11+
persistence_layer = DynamoDBPersistenceLayer(table_name=table)
912
config = IdempotencyConfig(
1013
event_key_jmespath='["user.uid", "order_id"]',
1114
raise_on_no_idempotency_key=True,

Diff for: ‎examples/idempotency/src/working_with_idempotent_function_custom_output_serializer.py

+3-1
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import os
12
from typing import Dict, Type
23

34
from aws_lambda_powertools.utilities.idempotency import (
@@ -8,7 +9,8 @@
89
from aws_lambda_powertools.utilities.idempotency.serialization.custom_dict import CustomDictSerializer
910
from aws_lambda_powertools.utilities.typing import LambdaContext
1011

11-
dynamodb = DynamoDBPersistenceLayer(table_name="IdempotencyTable")
12+
table = os.getenv("IDEMPOTENCY_TABLE", "")
13+
dynamodb = DynamoDBPersistenceLayer(table_name=table)
1214
config = IdempotencyConfig(event_key_jmespath="order_id") # see Choosing a payload subset section
1315

1416

0 commit comments

Comments
 (0)