Bug: etag is a required field in S3EventNotificationObjectModel however is not present in DeleteObject events

[benjamingorman]

Expected Behaviour

etag should be an Optional field in this pydantic schema

Current Behaviour

Good afternoon chaps.
I'm noticing what I think is an issue with one of the pydantic models in powertools

powertools-lambda-python/aws_lambda_powertools/utilities/parser/models/s3.py

Line 61 in c30bbdd

class S3EventNotificationObjectModel(BaseModel):

class S3EventNotificationObjectModel(BaseModel):
    key: str
    size: Optional[NonNegativeFloat] = None
    etag: str
    version_id: str = Field(None, alias="version-id")
    sequencer: Optional[str] = None

etag here is required but it seems that for S3 DeleteObject events, etag is sometimes not set:

For example, here's an excerpt of an event I received from S3 via eventbridge:

{
    "version": "0",
    "id": "5afe384b-7341-a76f-25ab-29bac47a969a",
    "detail-type": "Object Deleted",
    "source": "aws.s3",
    "account": "REDACTED",
    "time": "REDACTED",
    "region": "eu-west-1",
    "resources": [
        "arn:aws:s3:::my_bucket"
    ],
    "detail": {
        "version": "0",
        "bucket": {
            "name": "my_bucket"
        },
        "object": {
            "key": "REDACTED",
            "sequencer": "REDACTED"
        },
        "request-id": "REDACTED",
        "requester": "REDACTED",
        "source-ip-address": "REDACTED",
        "reason": "DeleteObject",
        "deletion-type": "Permanently Deleted"
    }
}

The pydantic schema is expecting detail.object.etag to be present here, however since this is a deletion event it doesn't exist and the event therefore seems invalid to pydantic
I think the resolution needed here is just to make etag an Optional field in the pydantic schema.
In my code I've tested this by making a custom version of S3EventNotificationObjectModel where etag is Optional and this does resolve my issue.

Code snippet

class MySqsRecordModel(SqsRecordModel):
    """Model for an individual record from the SQS event we received.

    We expect this to be an EventBridge event, which contains an S3 event.
    """

    body: Json[S3EventNotificationEventBridgeModel]


class MySqsEventModel(BaseModel):
    """Model for the SQS event we expect to receive."""

    Records: Sequence[MySqsRecordModel]

def lambda_handler(event: dict, context: LambdaContext) -> dict:
    """Handles the main entry point for the Lambda function."""
    logger.info("Received event", extra={"event": event, "context": context})

    # Attempt to parse the event according to the model we expect
    try:
        parsed_event: MySqsEventModel = parse(
            event=event, model=MySqsEventModel
        )
    except ValidationError as exc:
        logger.error("Failed to parse event", extra={"error": exc})
        raise

Possible Solution

Make etag an optional field in the pydantic schema so validation succeeds

Steps to Reproduce

In my own code I have an S3 bucket with events going to EventBridge. EventBridge forwards to SQS queue and lambda pulls from SQS queue.

All that's needed to observe the issue is to delete an object from the bucket, which creates a DeleteObject event. This event goes from S3 bucket -> EventBridge -> SQS Queue -> Lambda and the issue is visible in the lambda when it tries to parse the event according to schema in the code snippet.

Powertools for AWS Lambda (Python) version

2.37

AWS Lambda function runtime

3.10

Packaging format used

PyPi

Debugging logs

No response

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

[leandrodamascena]

Hey @benjamingorman! Thanks for opening this issue and reporting this bug.

Looks like we need to fix the problem in our Event Source Data Class we well: https://github.com/aws-powertools/powertools-lambda-python/blob/develop/aws_lambda_powertools/utilities/data_classes/s3_event.py#L28

Do you want to submit a PR to fix this? We'd love to have your first contribution here.

[heitorlessa]

Update: we're merging this shortly. We've got a few other fixes to get done and will look to make a release this or next week.

Thank you @benjamingorman for all the effort in reporting and creating a PR for this, we can't thank you enough!

For anyone reading later, what we've confirmed is:

• etag (EventBridge->S3) and eTag (S3->Lambda/SNS/SQS) are omitted for Object Deletion events
• size is omitted for Object Deletion events

What we've done to complete the PR:

• We fixed size return type to be int instead of `str
• We updated tests to assert the possibility of fields being optional
• We created a stack to be quadruple sure that's the case (real event below)

---

Object deletion omitting both etag and size

{
        "version": "0",
        "id": "46bf2718-5e27-a74d-ec6b-2d95d4e21844",
        "detail-type": "Object Deleted",
        "source": "aws.s3",
        "account": "536254204126",
        "time": "2024-05-07T09:31:52Z",
        "region": "eu-west-1",
        "resources": [
            "arn:aws:s3:::s3teststack-mybucketf68f3ff0-gscrrokfnvnw"
        ],
        "detail": {
            "version": "0",
            "bucket": {
                "name": "s3teststack-mybucketf68f3ff0-gscrrokfnvnw"
            },
            "object": {
                "key": "surprised-pikachu.gif",
                "sequencer": "006639F508B96A7690"
            },
            "request-id": "11K82SGF9X2HBEKD",
            "requester": "536254204126",
            "source-ip-address": "15.248.3.15",
            "reason": "DeleteObject",
            "deletion-type": "Permanently Deleted"
        }
    }

Object creation containing both etag and size

{
        "version": "0",
        "id": "52cc447b-01d6-df6f-8ae6-aea40b330218",
        "detail-type": "Object Created",
        "source": "aws.s3",
        "account": "536254204126",
        "time": "2024-05-07T09:25:51Z",
        "region": "eu-west-1",
        "resources": [
            "arn:aws:s3:::s3teststack-mybucketf68f3ff0-gscrrokfnvnw"
        ],
        "detail": {
            "version": "0",
            "bucket": {
                "name": "s3teststack-mybucketf68f3ff0-gscrrokfnvnw"
            },
            "object": {
                "key": "surprised-pikachu.gif",
                "size": 2638774,
                "etag": "fb21c5a0ff18e29aab890d1d1f6d6e64",
                "sequencer": "006639F39F56511781"
            },
            "request-id": "B5Z9SBP1A8R7DERF",
            "requester": "536254204126",
            "source-ip-address": "15.248.3.15",
            "reason": "PutObject"
        }
    }

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.

[github-actions]

This is now released under 2.38.0 version!