Bug: process_partial_response or processor does not catch raised errors in subroutine

[mhboon]

Expected Behaviour

Inline with the provided examples:
https://docs.powertools.aws.dev/lambda/python/latest/utilities/batch/#accessing-processed-messages and https://docs.powertools.aws.dev/lambda/python/latest/utilities/batch/#recommended

the expected behavior is that in case record_handler raises and exception. either processor or process_partial_response catches this for further controlled handling.

Current Behaviour

In case record_handler raises an exception, this results in a BatchProcessingError

Code snippet

def record_handler(record: SQSRecord):
    raise Exception("not ok")
def lambda_handler(event, context: LambdaContext):
    return process_partial_response(  
        event=event,
        record_handler=record_handler,
        processor=processor,
        context=context,
    )

Possible Solution

No response

Steps to Reproduce

Not yet tested when running in a lambda; reproduced when running locally.

Powertools for AWS Lambda (Python) version

latest

AWS Lambda function runtime

3.8

Packaging format used

Lambda Layers

Debugging logs

No response

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

[heitorlessa]

Hey @mhboon, that's the expected behaviour when all records in the batch fail to be processed.

https://docs.powertools.aws.dev/lambda/python/latest/utilities/batch/#partial-failure-mechanics

If you however have a partial failure, that is one successful amongst many failures, then it won't raise.

We do raise in total failure scenarios with all exceptions collected to help you diagnose an unwanted scenario. For poison pills, partial failures, etc. we handle just as fine.

Does that help? Or did you notice a regression and an exception is being raised despite a partial success?

[mhboon]

Got it; in that case, I misunderstood and will close the issue. Thanks for clarifying.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.