Bug: Retry of previously failed SQS record is skipped when skipGroupOnError is enabled

[mlrprananta]

Expected Behavior

When a SQS record that failed in a previous Lambda invocation is retried and handled by the same Lambda instance again, it should be processed by the provided recordHandler

Current Behavior

When a SQS record that failed in a previous Lambda batch invocation is retried and handled by the same Lambda instance again, it is skipped and returned as a failure without going through the recordHandler.

Only if the record is handled by a concurrent Lambda instance will it be processed again.

Code snippet

import { processPartialResponse, SqsFifoPartialProcessorAsync } from "@aws-lambda-powertools/batch";
import { SQSHandler, SQSRecord } from "aws-lambda";

const processor = new SqsFifoPartialProcessorAsync();

const recordHandler = async (record: SQSRecord): Promise<void> => {
    console.debug("RECORD HANDLER INVOKED");
    throw new Error("Random error occurred");
};

export const handle: SQSHandler = async (event, context) => {
    console.debug('SQS HANDLER INVOKED');
    return processPartialResponse(event, recordHandler, processor, {
        context,
        skipGroupOnError: true,
        throwOnFullBatchFailure: false,
    });
};

Steps to Reproduce

1. Set up a FIFO queue, a DLQ and a Lambda with the above handler code and set reserved concurrency to 0
2. Publish any message to the queue
3. In the logs you should observe SQS HANDLER INVOKED and RECORD HANDLER INVOKED for the first invocation, and after the visibility timeout it will only log SQS HANDLER INVOKED for follow-up invocations, until maxReceiveCount is hit and the message is moved to the DLQ

Possible Solution

In SqsProcessor, the failedGroupIds should be cleared after each Lambda invocation.

Introduce a method to clear the failedGroupIds in SqsProcesor.

In SqsFifoPartialProcessorAsync override the prepare method and add a call to clear failedGroupIds.

Powertools for AWS Lambda (TypeScript) version

latest

AWS Lambda function runtime

18.x

Packaging format used

npm

Execution logs

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #typescript channel on our Powertools for AWS Lambda Discord: Invite link

[dreamorosi]

Hi @mlrprananta, thank you for opening this issue and providing a potential solution.

I was able to reproduce the issue you described. I'll work on a PR to fix it.

[github-actions]

⚠️ COMMENT VISIBILITY WARNING ⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.

[github-actions]

This is now released under v2.16.0 version!