Feature request: Support for large SQS payloads in Event Data Source Classes

[plondino]

Use case

AWS has support for large payloads in SQS using the extended client library: https://pypi.org/project/sqs-extended-client/ - However there's still a fair amount of boilerplate when reacting to the event in a lambda on the receiving end to unwrap the message, check if it's a large payload message, and retrieve the payload from S3.

Solution/User Experience

This seems like an ideal use case for Powertools to reduce the boilerplate code and be able to unwrap an SQS, check for large payload support and retrieve the message as part of the receipt of the message.

Example payload:

{
    "Records": [
        {
            "messageId": "4bdc116c-dda9-4344-bfb1-cf49a2bcdd68",
            "receiptHandle": "",
            "body": "[\"com.amazon.sqs.javamessaging.MessageS3Pointer\",{\"s3BucketName\":\"bucket_name\",\"s3Key\":\"s3_key\"}]",
            "attributes": {
                "ApproximateReceiveCount": "1",
                "SentTimestamp": "1726071666910",
                "SequenceNumber": "18888618420438511616",
                "MessageGroupId": "d57f6826-89c2-497e-b920-ec6dc5dff067",
                "SenderId": "user_id:[email protected]",
                "MessageDeduplicationId": "d39dc4b9-52eb-425e-be3a-c57649d59740",
                "ApproximateFirstReceiveTimestamp": "1726071666910"
            },
            "messageAttributes": {
                "SQSLargePayloadSize": {
                    "stringValue": "300000",
                    "stringListValues": [],
                    "binaryListValues": [],
                    "dataType": "Number"
                }
            },
            "md5OfMessageAttributes": "5db28f0fb9aab47287f040ec3714a262",
            "md5OfBody": "d7b208d9230692ec7d4ab5588266024c",
            "eventSource": "aws:sqs",
            "eventSourceARN": "arn:aws:sqs:us-east-1:123456789012:sqs_name",
            "awsRegion": "us-east-1"
        }
    ]
}

Alternative solutions

No response

Acknowledgment

• This feature request meets Powertools for AWS Lambda (Python) Tenets
• Should this be considered in other Powertools for AWS Lambda languages? i.e. Java, TypeScript, and .NET

[sthulb]

@plondino Thanks for opening the issue – we have this in the Java version of Powertools for AWS Lambda already, so it seems like it would be a good fit for future releases.

We're currently paused for the v3 release next week, but we'll look into this when we can.

[shubhamsugara22]

@sthulb looks interesting can i try this feature request

[anafalcao]

Hey @shubhamsugara22 , we'll research more about this, and come back to you as soon as I have an update.

[leandrodamascena]

Hello @plondino and @shubhamsugara22! If I understand correctly, this sqs-extended-client library implements the same interface/idea as the https://github.com/awslabs/amazon-sqs-java-extended-client-lib library and makes it interoperable because the payload body contains the same point class as Java: com.amazon.sqs.javamessaging.MessageS3Pointer and s3BucketName/s3Key, right?

If this is true, what we could improve in this experience?

from aws_lambda_powertools.utilities.data_classes import SQSEvent, event_source
import boto3

s3_client = boto3.client("s3")

@event_source(data_class=SQSEvent)
def lambda_handler(event: SQSEvent, context):
  for record in my_event.records:
      body = record.json_body
  
      if "com.amazon.sqs.javamessaging.MessageS3Pointer" in body:
          s3_bucket_name = record.json_body[1]["s3BucketName"]
          s3_key = record.json_body[1]["s3Key"]
  
          s3_file = s3.get_object(Bucket="s3_bucket_name", Key=s3_key)

This seems like an ideal use case for Powertools to reduce the boilerplate code and be able to unwrap an SQS, check for large payload support and retrieve the message as part of the receipt of the message.

You mean that if the message container com.amazon.sqs.javamessaging.MessageS3Pointer in the body, we should automatically get the file from S3? But how do we return it? Inject it as the message body?

We currently support accessing the message body and getting the keys, maybe we can think of a class for that to access it as an object, but getting the file directly from S3 I would like to hear more about this idea.

Thanks

[anafalcao]

Since we haven't heard back, we'll be closing it for now. Please don't hesitate to open a new one or start a new discussion if you have any updates or further thoughts. We appreciate your input!

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.