Feature request: Improve error message when parser fails

[Mavtti]

Expected Behaviour

I would expect it to work out of the box

Current Behaviour

I've wrapped my function with event_parser for a kinesis input. For some records and not all, I get the following error:

[ERROR] InvalidEnvelopeError: Envelope must implement BaseEnvelope, envelope=<class 'aws_lambda_powertools.utilities.parser.envelopes.kinesis.KinesisDataStreamEnvelope'>
Traceback (most recent call last):
  File "/var/task/aws_lambda_powertools/middleware_factory/factory.py", line 135, in wrapper
    response = middleware()
  File "/var/task/aws_lambda_powertools/utilities/parser/parser.py", line 98, in event_parser
    parsed_event = parse(event=event, model=model, envelope=envelope)
  File "/var/task/aws_lambda_powertools/utilities/parser/parser.py", line 175, in parse
    raise InvalidEnvelopeError(f"Envelope must implement BaseEnvelope, envelope={envelope}")

I've tested to manually call KinesisDataStreamEnvelope.parse and it works fine.

Strange thing, it doesn't happen for every lambda invocation.

Thx

Code snippet

from aws_lambda_powertools.utilities.parser import envelopes
from aws_lambda_powertools.utilities.parser import event_parser
from aws_lambda_powertools.utilities.typing import LambdaContext


@event_parser(
    model=SuggestionPayload,
    envelope=envelopes.KinesisDataStreamEnvelope,
)
def lambda_handler(document_events: list[SuggestionPayload], _: LambdaContext):

Possible Solution

No response

Steps to Reproduce

Lambda with Kinesis data stream connector and the previous code snippet.
Lambda is in a Docker format and not zip.

Powertools for AWS Lambda (Python) version

latest

AWS Lambda function runtime

3.10

Packaging format used

PyPi

Debugging logs

No debug logs before this error

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

[leandrodamascena]

Looking at this now..

[leandrodamascena]

Hi @Mavtti! Thanks for opening this issue to report a possible bug. Just to confirm my suspicion: your SuggestionPayload doesn't extend from BaseModel, right? Here we parse the envelope using the provided Model (SuggestionPayload) and both (Model and Envelope) must extend from the BaseModel, otherwise the parse will fail.

However, I think there is room for us to improve our error message, as it leads us to believe that it is a problem with the Envelope and not the Model.

What do you think about sending a PR and improving the error message?

Thanks

---

Example of code that works

from aws_lambda_powertools.utilities.parser import envelopes
from aws_lambda_powertools.utilities.parser.models import KinesisDataStreamModel
from aws_lambda_powertools.utilities.parser import event_parser
from aws_lambda_powertools.utilities.typing import LambdaContext

from pydantic import BaseModel

class SuggestionPayload(BaseModel):
    message: str
    username: str


@event_parser(model=SuggestionPayload, envelope=envelopes.KinesisDataStreamEnvelope)
def lambda_handler(document_events: KinesisDataStreamModel, _: LambdaContext):
    print(document_events)

Payload

{
  "Records": [
    {
      "kinesis": {
        "kinesisSchemaVersion": "1.0",
        "partitionKey": "1",
        "sequenceNumber": "49590338271490256608559692538361571095921575989136588898",
        "data": "eyJtZXNzYWdlIjogInRlc3QgbWVzc2FnZSIsICJ1c2VybmFtZSI6ICJ0ZXN0In0=",
        "approximateArrivalTimestamp": 1545084650.987
      },
      "eventSource": "aws:kinesis",
      "eventVersion": "1.0",
      "eventID": "shardId-000000000006:49590338271490256608559692538361571095921575989136588898",
      "eventName": "aws:kinesis:record",
      "invokeIdentityArn": "arn:aws:iam::123456789012:role/lambda-role",
      "awsRegion": "us-east-2",
      "eventSourceARN": "arn:aws:kinesis:us-east-2:123456789012:stream/lambda-stream"
    }
  ]
}

data field decoded

echo eyJtZXNzYWdlIjogInRlc3QgbWVzc2FnZSIsICJ1c2VybmFtZSI6ICJ0ZXN0In0= |base64 -di
{"message": "test message", "username": "test"}

[Mavtti]

Hey @leandrodamascena thank for looking into it.

Actually my SuggestionPayload does inherit from BaseModel.

Here is a sample:

from pydantic import BaseModel

from .document import Document


class SuggestionPayload(BaseModel):
    arrived_at: Optional[str | int | float]
    data: Document
    operation: str

And the subclass Document also inherit from BaseModel.

I've tried using:
KinesisDataStreamEnvelope().parse(...) instead with no issue.

[leandrodamascena]

Hi @Mavtti! Sorry to hear this, but in all my attempts here I could not reproduce the error. I am interested in understanding what is going wrong and fixing it.
Can you please send me a payload that you are using to invoke your Lambda? Please remove any sensitive data.

Thank you

[Mavtti]

Hey @leandrodamascena in the end I've been able to reproduce on my side.

You were right on the displayed error message, it hides the real issue. For my case it came from an AttributeError on the SuggestionPayload class.

And this AttributeError was catched by the try/except. If it was any other error it would have been more explicit.

Thx for the help.

[leandrodamascena]

Hey @Mavtti! Nice to hear you could figure out this. I sent a PullRequest to improve the error message and make it more clear.

Thanks again for taking the time to help us improve the developer experience. I will give you credits when I release a version with this fix.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.

[github-actions]

This is now released under 2.31.0 version!