Feature request: Report batch item failures for consuming message from MSK

[yifei-ding]

Use case

Customer is handling IoT data using MSK and lambda. They work in highly regulated industry so they need to ensure there's no missing data. They want to know which item is failed in a batch in order to process it again. Currently this function is available with SQS but not with MSK yet. https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html#services-sqs-batchfailurereporting

Solution/User Experience

When consuming a batch of message from MSK using lambda function, return the failed item id(s).

Alternative solutions

No response

Acknowledgment

• This feature request meets Powertools for AWS Lambda (Python) Tenets
• Should this be considered in other Powertools for AWS Lambda languages? i.e. Java, TypeScript, and .NET

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

[leandrodamascena]

Hello @yifei-ding! Good to know that companies are using MSK to store data for IoT and want to take advantage of adding partial failure support. But at this moment Lambda currently doesn't support this feature - as you mentioned.

We have a similar open issue (#2969) where other customers are also requesting this support. We are collecting these requests to pass the needs of our customers to the Lambda team, but we are unable to give an estimate of when we can update this issue with any updates.

Do you mind if I close this issue and you give a thumbs-up/add a comment on issue #2969?

Thank you.

[yifei-ding]

Sure. Please go ahead and close this issue.

Hello @yifei-ding! Good to know that companies are using MSK to store data for IoT and want to take advantage of adding partial failure support. But at this moment Lambda currently doesn't support this feature - as you mentioned.

We have a similar open issue (#2969) where other customers are also requesting this support. We are collecting these requests to pass the needs of our customers to the Lambda team, but we are unable to give an estimate of when we can update this issue with any updates.

Do you mind if I close this issue and you give a thumbs-up/add a comment on issue #2969?

Thank you.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.