Skip to content

feat(jmespath): new built-in envelopes to unwrap S3 events #2169

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
May 1, 2023
Merged

feat(jmespath): new built-in envelopes to unwrap S3 events #2169

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
ef81b2f
feat(jmespath): adding new envelopes + docs + small fix
leandrodamascena Apr 26, 2023
0de8370
feat(jmespath): text
leandrodamascena Apr 27, 2023
b13a6f9
Merge branch 'develop' into jmespath/envelopes
leandrodamascena Apr 27, 2023
7528d07
chore: addressing Ruben's feedback
leandrodamascena Apr 27, 2023
a5152dc
feat(jmespath): addressing Heitor's feedback
leandrodamascena Apr 28, 2023
ef54196
chore: break message in two sentences.
heitorlessa May 1, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions aws_lambda_powertools/utilities/jmespath_utils/envelopes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,8 @@
CLOUDWATCH_EVENTS_SCHEDULED = EVENTBRIDGE
KINESIS_DATA_STREAM = "Records[*].kinesis.powertools_json(powertools_base64(data))"
CLOUDWATCH_LOGS = "awslogs.powertools_base64_gzip(data) | powertools_json(@).logEvents[*]"
S3_SNS_SQS = "Records[*].powertools_json(body).powertools_json(Message).Records[0]"
S3_SQS = "Records[*].powertools_json(body).Records[0]"
S3_SNS_KINESIS_FIREHOSE = "records[*].powertools_json(powertools_base64(data)).powertools_json(Message).Records[0]"
S3_KINESIS_FIREHOSE = "records[*].powertools_json(powertools_base64(data)).Records[0]"
S3_EVENTBRIDGE_SQS = "Records[*].powertools_json(body).detail"
28 changes: 18 additions & 10 deletions docs/utilities/jmespath_functions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,16 +64,24 @@ We provide built-in envelopes for popular AWS Lambda event sources to easily dec

These are all built-in envelopes you can use along with their expression as a reference:

| Envelope | JMESPath expression |
| --------------------------------- | ------------------------------------------------------------- |
| **`API_GATEWAY_REST`** | `powertools_json(body)` |
| **`API_GATEWAY_HTTP`** | `API_GATEWAY_REST` |
| **`SQS`** | `Records[*].powertools_json(body)` |
| **`SNS`** | `Records[0].Sns.Message | powertools_json(@)` |
| **`EVENTBRIDGE`** | `detail` |
| **`CLOUDWATCH_EVENTS_SCHEDULED`** | `EVENTBRIDGE` |
| **`KINESIS_DATA_STREAM`** | `Records[*].kinesis.powertools_json(powertools_base64(data))` |
| **`CLOUDWATCH_LOGS`** | `awslogs.powertools_base64_gzip(data) | powertools_json(@).logEvents[*]` |
| Envelope | JMESPath expression |
| --------------------------------- | ------------------------------------------------------------------------------------------ |
| **`API_GATEWAY_HTTP`** | `powertools_json(body)` |
| **`API_GATEWAY_REST`** | `powertools_json(body)` |
| **`CLOUDWATCH_EVENTS_SCHEDULED`** | `detail` |
| **`CLOUDWATCH_LOGS`** | `awslogs.powertools_base64_gzip(data) | powertools_json(@).logEvents[*]` |
| **`EVENTBRIDGE`** | `detail` |
| **`KINESIS_DATA_STREAM`** | `Records[*].kinesis.powertools_json(powertools_base64(data))` |
| **`S3_EVENTBRIDGE_SQS`** | `Records[*].powertools_json(body).detail` |
| **`S3_KINESIS_FIREHOSE`** | `records[*].powertools_json(powertools_base64(data)).Records[0]` |
| **`S3_SNS_KINESIS_FIREHOSE`** | `records[*].powertools_json(powertools_base64(data)).powertools_json(Message).Records[0]` |
| **`S3_SNS_SQS`** | `Records[*].powertools_json(body).powertools_json(Message).Records[0]` |
| **`S3_SQS`** | `Records[*].powertools_json(body).Records[0]` |
| **`SNS`** | `Records[0].Sns.Message | powertools_json(@)` |
| **`SQS`** | `Records[*].powertools_json(body)` |

???+ tip "Using SNS?"
Consider reducing the payload size by enabling the [raw message delivery](https://docs.aws.amazon.com/sns/latest/dg/sns-large-payload-raw-message-delivery.html){target="_blank"} option in Amazon SNS if you're utilizing architectures that require sending event notifications from S3 to SNS and then to SQS or Kinesis Data Firehose, but do not require the SNS metadata.

## Advanced

Expand Down
20 changes: 10 additions & 10 deletions docs/utilities/validation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,16 +141,16 @@ We provide built-in envelopes to easily extract the payload from popular event s

Here is a handy table with built-in envelopes along with their JMESPath expressions in case you want to build your own.

| Envelope name | JMESPath expression |
| ------------------------------- | ------------------------------------------------------------- |
| **API_GATEWAY_REST** | "powertools_json(body)" |
| **API_GATEWAY_HTTP** | "powertools_json(body)" |
| **SQS** | "Records[*].powertools_json(body)" |
| **SNS** | "Records[0].Sns.Message | powertools_json(@)" |
| **EVENTBRIDGE** | "detail" |
| **CLOUDWATCH_EVENTS_SCHEDULED** | "detail" |
| **KINESIS_DATA_STREAM** | "Records[*].kinesis.powertools_json(powertools_base64(data))" |
| **CLOUDWATCH_LOGS** | "awslogs.powertools_base64_gzip(data) | powertools_json(@).logEvents[*]" |
| Envelope | JMESPath expression |
| --------------------------------- | ------------------------------------------------------------------------ |
| **`API_GATEWAY_HTTP`** | `powertools_json(body)` |
| **`API_GATEWAY_REST`** | `powertools_json(body)` |
| **`CLOUDWATCH_EVENTS_SCHEDULED`** | `detail` |
| **`CLOUDWATCH_LOGS`** | `awslogs.powertools_base64_gzip(data) | powertools_json(@).logEvents[*]` |
| **`EVENTBRIDGE`** | `detail` |
| **`KINESIS_DATA_STREAM`** | `Records[*].kinesis.powertools_json(powertools_base64(data))` |
| **`SNS`** | `Records[0].Sns.Message | powertools_json(@)` |
| **`SQS`** | `Records[*].powertools_json(body)` |

## Advanced

Expand Down
12 changes: 9 additions & 3 deletions examples/jmespath_functions/src/extract_data_from_builtin_envelope.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,18 @@
from __future__ import annotations

from aws_lambda_powertools import Logger
from aws_lambda_powertools.utilities.jmespath_utils import (
envelopes,
extract_data_from_envelope,
)
from aws_lambda_powertools.utilities.typing import LambdaContext

logger = Logger()


def handler(event: dict, context: LambdaContext) -> dict:
payload = extract_data_from_envelope(data=event, envelope=envelopes.SQS)
customer_id = payload.get("customerId") # now deserialized
records: list = extract_data_from_envelope(data=event, envelope=envelopes.SQS)
for record in records: # records is a list
logger.info(record.get("customerId")) # now deserialized

return {"customer_id": customer_id, "message": "success", "statusCode": 200}
return {"message": "success", "statusCode": 200}