Skip to content

feat(event_source): support custom json_deserializer; add json_body in SQSEvent #2200

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

leandrodamascena
Copy link
Contributor

Issue number: #2199

Summary

Changes

This PR aims to add support for a custom json deserialization method in the event source classes.

User experience

BEFORE

from aws_lambda_powertools.utilities.data_classes import event_source, SQSEvent
import json 

def lambda_handler(event: SQSEvent, context):
    parsed = SQSEvent(data=event)
    # Multiple records can be delivered in a single event
    for record in parsed.records:
        print(json.loads(record.body))

AFTER

from aws_lambda_powertools.utilities.data_classes import event_source, SQSEvent
import ujson

def lambda_handler(event: SQSEvent, context):
    
    parsed = SQSEvent(data = event, json_deserializer=ujson.loads)
    # Multiple records can be delivered in a single event
    for record in parsed.records:
        print(record.json_body)

Checklist

If your change doesn't seem to apply, please leave them unchecked.

Is this a breaking change?

RFC issue number:

Checklist:

  • Migration process documented
  • Implement warnings (if it can live side by side)

Acknowledgment

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

@leandrodamascena leandrodamascena requested a review from a team as a code owner May 3, 2023 14:23
@leandrodamascena leandrodamascena requested review from heitorlessa and removed request for a team May 3, 2023 14:23
@boring-cyborg boring-cyborg bot added the tests label May 3, 2023
@pull-request-size pull-request-size bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label May 3, 2023
@leandrodamascena leandrodamascena linked an issue May 3, 2023 that may be closed by this pull request
2 tasks
@github-actions github-actions bot added the feature New feature or functionality label May 3, 2023
Comment on lines 106 to 114
@property
def json_body(self) -> Dict:
"""Parses the submitted body as json"""
try:
if self._json_data is None:
self._json_data = self._json_deserializer(self["body"])
return self._json_data
except Exception:
return self["body"]
Copy link
Contributor Author

@leandrodamascena leandrodamascena May 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason I add a try/except block here is because it could be a simple string that cannot be deserialized as json.
Should we consider this for the other json_body fields in Kinesis/Kafka/APIGW? Should we return the plain body if the deserialization fails?
I would like to hear from you.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should fail hard here to give the caller a chance to handle this error correctly.

Reasoning in a mental model fashion:

  • Do we offer a non-JSON body property e.g., body?
    • If so, json_body should fail if it can't deserialize it
  • If the customer received a poison pill record, a non-JSON string, will they be given the chance to handle a JSON deserialization error?
    • If we return self["body"], it can result to a totally different error we can't even anticipate, e.g., json_body["blah"]

As a rule of thumb, you'd want to propagate the error as close to the actual problem so they instinctively know how to handle it. If we catch all, we're masking the actual error and leading them to an unknown error later on.

tiny note: until Python 3.11, try/catch has a performance overhead even if it's tiny, so we can also improve here by letting them do it (not accidentally twice).

@codecov-commenter
Copy link

codecov-commenter commented May 3, 2023

Codecov Report

Patch coverage: 96.15% and project coverage change: -0.01 ⚠️

Comparison is base (32666e4) 97.47% compared to head (708a7e6) 97.46%.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #2200      +/-   ##
===========================================
- Coverage    97.47%   97.46%   -0.01%     
===========================================
  Files          148      149       +1     
  Lines         6882     6895      +13     
  Branches       505      506       +1     
===========================================
+ Hits          6708     6720      +12     
  Misses         137      137              
- Partials        37       38       +1     
Impacted Files Coverage Δ
...bda_powertools/utilities/data_classes/sqs_event.py 98.85% <85.71%> (-1.15%) ⬇️
...lambda_powertools/utilities/data_classes/common.py 100.00% <100.00%> (ø)
...a_powertools/utilities/data_classes/kafka_event.py 92.64% <100.00%> (-0.11%) ⬇️
...s/utilities/data_classes/kinesis_firehose_event.py 98.41% <100.00%> (-0.03%) ⬇️
...bda_powertools/utilities/parser/models/__init__.py 100.00% <100.00%> (ø)
...s/utilities/parser/models/s3_event_notification.py 100.00% <100.00%> (ø)
...s_lambda_powertools/utilities/parser/models/sqs.py 100.00% <100.00%> (ø)

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.


from aws_lambda_powertools.shared.headers_serializer import BaseHeadersSerializer


class DictWrapper(Mapping):
"""Provides a single read only access to a wrapper dict"""

def __init__(self, data: Dict[str, Any]):
def __init__(self, data: Dict[str, Any], json_deserializer: Optional[Callable] = None):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

docstring ;) we apparently missed back in the days for DictWrapper, but given the json_deserializer will be added to all data classes, we can propagate the docstring to everyone benefits from it.

pushing that change

image

@pull-request-size pull-request-size bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels May 4, 2023
@heitorlessa heitorlessa changed the title feat(event_source): add support to custom json decode in event class feat(event_source): support custom json_deserializer; add json_body in SQSEvent May 4, 2023
@leandrodamascena leandrodamascena merged commit 3660dbf into aws-powertools:develop May 4, 2023
@leandrodamascena leandrodamascena deleted the feat/decodejson branch May 4, 2023 12:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or functionality size/L Denotes a PR that changes 100-499 lines, ignoring generated files. tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Feature request: support custom JSON deserializer in event source classes
3 participants