Skip to content

Commit 204eb98

Browse files
walmslesto-mc
walmsles
and
to-mc
authored
docs(idempotency): fix misleading idempotent examples (#661)
Co-authored-by: Tom McCarthy <[email protected]>
1 parent 977162a commit 204eb98

File tree

4 files changed

+176
-125
lines changed

4 files changed

+176
-125
lines changed

‎docs/utilities/idempotency.md

+8-2
Original file line numberDiff line numberDiff line change
@@ -206,6 +206,11 @@ In this example, we have a Lambda handler that creates a payment for a user subs
206206

207207
Imagine the function executes successfully, but the client never receives the response due to a connection issue. It is safe to retry in this instance, as the idempotent decorator will return a previously saved response.
208208

209+
!!! warning "Idempotency for JSON payloads"
210+
The payload extracted by the `event_key_jmespath` is treated as a string by default, so will be sensitive to differences in whitespace even when the JSON payload itself is identical.
211+
212+
To alter this behaviour, we can use the [JMESPath built-in function](/utilities/jmespath_functions) *powertools_json()* to treat the payload as a JSON object rather than a string.
213+
209214
=== "payment.py"
210215

211216
```python hl_lines="2-4 10 12 15 20"
@@ -218,7 +223,7 @@ Imagine the function executes successfully, but the client never receives the re
218223

219224
# Treat everything under the "body" key
220225
# in the event json object as our payload
221-
config = IdempotencyConfig(event_key_jmespath="body")
226+
config = IdempotencyConfig(event_key_jmespath="powertools_json(body)")
222227

223228
@idempotent(config=config, persistence_store=persistence_layer)
224229
def handler(event, context):
@@ -270,6 +275,7 @@ Imagine the function executes successfully, but the client never receives the re
270275
}
271276
```
272277

278+
273279
### Idempotency request flow
274280

275281
This sequence diagram shows an example flow of what happens in the payment scenario:
@@ -334,7 +340,7 @@ Idempotent decorator can be further configured with **`IdempotencyConfig`** as s
334340

335341
Parameter | Default | Description
336342
------------------------------------------------- | ------------------------------------------------- | ---------------------------------------------------------------------------------
337-
**event_key_jmespath** | `""` | JMESPath expression to extract the idempotency key from the event record
343+
**event_key_jmespath** | `""` | JMESPath expression to extract the idempotency key from the event record using [built-in functions](/utilities/jmespath_functions)
338344
**payload_validation_jmespath** | `""` | JMESPath expression to validate whether certain parameters have changed in the event while the event payload
339345
**raise_on_no_idempotency_key** | `False` | Raise exception if no idempotency key was found in the request
340346
**expires_after_seconds** | 3600 | The number of seconds to wait before a record is expired

‎docs/utilities/jmespath_functions.md

+166
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,166 @@
1+
---
2+
title: JMESPath Functions
3+
description: Utility
4+
---
5+
6+
You might have events or responses that contain non-encoded JSON, where you need to decode so that you can access portions of the object or ensure the Powertools utility receives a JSON object. This is a common use case when using the [validation](/utilities/validation) or [idempotency](/utilities/idempotency) utilities.
7+
8+
## Built-in JMESPath functions
9+
You can use our built-in JMESPath functions within your expressions to do exactly that to decode JSON Strings, base64, and uncompress gzip data.
10+
11+
!!! info
12+
We use these for built-in envelopes to easily decode and unwrap events from sources like API Gateway, Kinesis, CloudWatch Logs, etc.
13+
14+
#### powertools_json function
15+
16+
Use `powertools_json` function to decode any JSON String anywhere a JMESPath expression is allowed.
17+
18+
> **Validation scenario**
19+
20+
This sample will decode the value within the `data` key into a valid JSON before we can validate it.
21+
22+
=== "powertools_json_jmespath_function.py"
23+
24+
```python hl_lines="9"
25+
from aws_lambda_powertools.utilities.validation import validate
26+
27+
import schemas
28+
29+
sample_event = {
30+
'data': '{"payload": {"message": "hello hello", "username": "blah blah"}}'
31+
}
32+
33+
validate(event=sample_event, schema=schemas.INPUT, envelope="powertools_json(data)")
34+
```
35+
36+
=== "schemas.py"
37+
38+
```python hl_lines="7 14 16 23 39 45 47 52"
39+
--8<-- "docs/shared/validation_basic_jsonschema.py"
40+
```
41+
42+
> **Idempotency scenario**
43+
44+
This sample will decode the value within the `body` key of an API Gateway event into a valid JSON object to ensure the Idempotency utility processes a JSON object instead of a string.
45+
46+
=== "powertools_json_jmespath_function.py"
47+
48+
```python hl_lines="8"
49+
import json
50+
from aws_lambda_powertools.utilities.idempotency import (
51+
IdempotencyConfig, DynamoDBPersistenceLayer, idempotent
52+
)
53+
54+
persistence_layer = DynamoDBPersistenceLayer(table_name="IdempotencyTable")
55+
56+
config = IdempotencyConfig(event_key_jmespath="powertools_json(body)")
57+
@idempotent(config=config, persistence_store=persistence_layer)
58+
def handler(event:APIGatewayProxyEvent, context):
59+
body = json.loads(event['body'])
60+
payment = create_subscription_payment(
61+
user=body['user'],
62+
product=body['product_id']
63+
)
64+
...
65+
return {
66+
"payment_id": payment.id,
67+
"message": "success",
68+
"statusCode": 200
69+
}
70+
```
71+
72+
#### powertools_base64 function
73+
74+
Use `powertools_base64` function to decode any base64 data.
75+
76+
This sample will decode the base64 value within the `data` key, and decode the JSON string into a valid JSON before we can validate it.
77+
78+
=== "powertools_json_jmespath_function.py"
79+
80+
```python hl_lines="12"
81+
from aws_lambda_powertools.utilities.validation import validate
82+
83+
import schemas
84+
85+
sample_event = {
86+
"data": "eyJtZXNzYWdlIjogImhlbGxvIGhlbGxvIiwgInVzZXJuYW1lIjogImJsYWggYmxhaCJ9="
87+
}
88+
89+
validate(
90+
event=sample_event,
91+
schema=schemas.INPUT,
92+
envelope="powertools_json(powertools_base64(data))"
93+
)
94+
```
95+
96+
=== "schemas.py"
97+
98+
```python hl_lines="7 14 16 23 39 45 47 52"
99+
--8<-- "docs/shared/validation_basic_jsonschema.py"
100+
```
101+
102+
#### powertools_base64_gzip function
103+
104+
Use `powertools_base64_gzip` function to decompress and decode base64 data.
105+
106+
This sample will decompress and decode base64 data, then use JMESPath pipeline expression to pass the result for decoding its JSON string.
107+
108+
=== "powertools_json_jmespath_function.py"
109+
110+
```python hl_lines="12"
111+
from aws_lambda_powertools.utilities.validation import validate
112+
113+
import schemas
114+
115+
sample_event = {
116+
"data": "H4sIACZAXl8C/52PzUrEMBhFX2UILpX8tPbHXWHqIOiq3Q1F0ubrWEiakqTWofTdTYYB0YWL2d5zvnuTFellBIOedoiyKH5M0iwnlKH7HZL6dDB6ngLDfLFYctUKjie9gHFaS/sAX1xNEq525QxwFXRGGMEkx4Th491rUZdV3YiIZ6Ljfd+lfSyAtZloacQgAkqSJCGhxM6t7cwwuUGPz4N0YKyvO6I9WDeMPMSo8Z4Ca/kJ6vMEYW5f1MX7W1lVxaG8vqX8hNFdjlc0iCBBSF4ERT/3Pl7RbMGMXF2KZMh/C+gDpNS7RRsp0OaRGzx0/t8e0jgmcczyLCWEePhni/23JWalzjdu0a3ZvgEaNLXeugEAAA=="
117+
}
118+
119+
validate(
120+
event=sample_event,
121+
schema=schemas.INPUT,
122+
envelope="powertools_base64_gzip(data) | powertools_json(@)"
123+
)
124+
```
125+
126+
=== "schemas.py"
127+
128+
```python hl_lines="7 14 16 23 39 45 47 52"
129+
--8<-- "docs/shared/validation_basic_jsonschema.py"
130+
```
131+
132+
### Bring your own JMESPath function
133+
134+
!!! warning
135+
This should only be used for advanced use cases where you have special formats not covered by the built-in functions.
136+
137+
This will **replace all provided built-in functions such as `powertools_json`, so you will no longer be able to use them**.
138+
139+
For special binary formats that you want to decode before applying JSON Schema validation, you can bring your own [JMESPath function](https://github.com/jmespath/jmespath.py#custom-functions){target="_blank"} and any additional option via `jmespath_options` param.
140+
141+
=== "custom_jmespath_function.py"
142+
143+
```python hl_lines="2 6-10 14"
144+
from aws_lambda_powertools.utilities.validation import validator
145+
from jmespath import functions
146+
147+
import schemas
148+
149+
class CustomFunctions(functions.Functions):
150+
151+
@functions.signature({'types': ['string']})
152+
def _func_special_decoder(self, s):
153+
return my_custom_decoder_logic(s)
154+
155+
custom_jmespath_options = {"custom_functions": CustomFunctions()}
156+
157+
@validator(schema=schemas.INPUT, jmespath_options=**custom_jmespath_options)
158+
def handler(event, context):
159+
return event
160+
```
161+
162+
=== "schemas.py"
163+
164+
```python hl_lines="7 14 16 23 39 45 47 52"
165+
--8<-- "docs/shared/validation_basic_jsonschema.py"
166+
```

‎docs/utilities/validation.md

+1-123
Original file line numberDiff line numberDiff line change
@@ -429,129 +429,7 @@ For each format defined in a dictionary key, you must use a regex, or a function
429429

430430
You might have events or responses that contain non-encoded JSON, where you need to decode before validating them.
431431

432-
You can use our built-in JMESPath functions within your expressions to do exactly that to decode JSON Strings, base64, and uncompress gzip data.
432+
You can use our built-in [JMESPath functions](/utilities/jmespath_functions) within your expressions to do exactly that to decode JSON Strings, base64, and uncompress gzip data.
433433

434434
!!! info
435435
We use these for built-in envelopes to easily to decode and unwrap events from sources like Kinesis, CloudWatch Logs, etc.
436-
437-
#### powertools_json function
438-
439-
Use `powertools_json` function to decode any JSON String.
440-
441-
This sample will decode the value within the `data` key into a valid JSON before we can validate it.
442-
443-
=== "powertools_json_jmespath_function.py"
444-
445-
```python hl_lines="9"
446-
from aws_lambda_powertools.utilities.validation import validate
447-
448-
import schemas
449-
450-
sample_event = {
451-
'data': '{"payload": {"message": "hello hello", "username": "blah blah"}}'
452-
}
453-
454-
validate(event=sample_event, schema=schemas.INPUT, envelope="powertools_json(data)")
455-
```
456-
457-
=== "schemas.py"
458-
459-
```python hl_lines="7 14 16 23 39 45 47 52"
460-
--8<-- "docs/shared/validation_basic_jsonschema.py"
461-
```
462-
463-
#### powertools_base64 function
464-
465-
Use `powertools_base64` function to decode any base64 data.
466-
467-
This sample will decode the base64 value within the `data` key, and decode the JSON string into a valid JSON before we can validate it.
468-
469-
=== "powertools_json_jmespath_function.py"
470-
471-
```python hl_lines="12"
472-
from aws_lambda_powertools.utilities.validation import validate
473-
474-
import schemas
475-
476-
sample_event = {
477-
"data": "eyJtZXNzYWdlIjogImhlbGxvIGhlbGxvIiwgInVzZXJuYW1lIjogImJsYWggYmxhaCJ9="
478-
}
479-
480-
validate(
481-
event=sample_event,
482-
schema=schemas.INPUT,
483-
envelope="powertools_json(powertools_base64(data))"
484-
)
485-
```
486-
487-
=== "schemas.py"
488-
489-
```python hl_lines="7 14 16 23 39 45 47 52"
490-
--8<-- "docs/shared/validation_basic_jsonschema.py"
491-
```
492-
493-
#### powertools_base64_gzip function
494-
495-
Use `powertools_base64_gzip` function to decompress and decode base64 data.
496-
497-
This sample will decompress and decode base64 data, then use JMESPath pipeline expression to pass the result for decoding its JSON string.
498-
499-
=== "powertools_json_jmespath_function.py"
500-
501-
```python hl_lines="12"
502-
from aws_lambda_powertools.utilities.validation import validate
503-
504-
import schemas
505-
506-
sample_event = {
507-
"data": "H4sIACZAXl8C/52PzUrEMBhFX2UILpX8tPbHXWHqIOiq3Q1F0ubrWEiakqTWofTdTYYB0YWL2d5zvnuTFellBIOedoiyKH5M0iwnlKH7HZL6dDB6ngLDfLFYctUKjie9gHFaS/sAX1xNEq525QxwFXRGGMEkx4Th491rUZdV3YiIZ6Ljfd+lfSyAtZloacQgAkqSJCGhxM6t7cwwuUGPz4N0YKyvO6I9WDeMPMSo8Z4Ca/kJ6vMEYW5f1MX7W1lVxaG8vqX8hNFdjlc0iCBBSF4ERT/3Pl7RbMGMXF2KZMh/C+gDpNS7RRsp0OaRGzx0/t8e0jgmcczyLCWEePhni/23JWalzjdu0a3ZvgEaNLXeugEAAA=="
508-
}
509-
510-
validate(
511-
event=sample_event,
512-
schema=schemas.INPUT,
513-
envelope="powertools_base64_gzip(data) | powertools_json(@)"
514-
)
515-
```
516-
517-
=== "schemas.py"
518-
519-
```python hl_lines="7 14 16 23 39 45 47 52"
520-
--8<-- "docs/shared/validation_basic_jsonschema.py"
521-
```
522-
523-
### Bring your own JMESPath function
524-
525-
!!! warning
526-
This should only be used for advanced use cases where you have special formats not covered by the built-in functions.
527-
528-
This will **replace all provided built-in functions such as `powertools_json`, so you will no longer be able to use them**.
529-
530-
For special binary formats that you want to decode before applying JSON Schema validation, you can bring your own [JMESPath function](https://github.com/jmespath/jmespath.py#custom-functions){target="_blank"} and any additional option via `jmespath_options` param.
531-
532-
=== "custom_jmespath_function.py"
533-
534-
```python hl_lines="2 6-10 14"
535-
from aws_lambda_powertools.utilities.validation import validator
536-
from jmespath import functions
537-
538-
import schemas
539-
540-
class CustomFunctions(functions.Functions):
541-
542-
@functions.signature({'types': ['string']})
543-
def _func_special_decoder(self, s):
544-
return my_custom_decoder_logic(s)
545-
546-
custom_jmespath_options = {"custom_functions": CustomFunctions()}
547-
548-
@validator(schema=schemas.INPUT, jmespath_options=**custom_jmespath_options)
549-
def handler(event, context):
550-
return event
551-
```
552-
553-
=== "schemas.py"
554-
555-
```python hl_lines="7 14 16 23 39 45 47 52"
556-
--8<-- "docs/shared/validation_basic_jsonschema.py"
557-
```

‎mkdocs.yml

+1
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@ nav:
2626
- utilities/parser.md
2727
- utilities/idempotency.md
2828
- utilities/feature_flags.md
29+
- utilities/jmespath_functions.md
2930

3031
theme:
3132
name: material

0 commit comments

Comments
 (0)