How is partial batch failure reporting designed to work for DyamoDB Streams?

[driimus]

I'm currently trying to port the batch processing utilities to TypeScript by borrowing some of the features found here, and some from the Java Powertools.

Everything has been relatively smooth so far, except for one bit: collecting DynamoDB failures.

I'm looking to gain a better understanding of the way it happens in the Python Powertools:
https://github.com/awslabs/aws-lambda-powertools-python/blob/63637978ba4f62493125b720abd882421a691291/aws_lambda_powertools/utilities/batch/base.py#L465

Given that dynamodb and SequenceNumber are described as an optional fields, I see two possibilities:

1. The sequence number is indeed optional, in which case a failure with no identifier will cause Lambda to treat the entire batch as a failure.
2. The expectation is that sequence numbers are always present, thus making it redundant to check it for truthiness before appending associated failures.

My TS implementation replicates the same behavior by falling back to an empty string, but I'm curious as to which scenario is at play.

[heitorlessa]

Hey, the sequence number is always present AFAIK as we're iterating on a shard when polled by Lambda poller. Have you experienced it otherwise when deployed?

The doc you linked is the API Data type, and many fields are sent despite having "Required: No" -- like "dynamodb" key, so I'm not sure why that is.

Another note is that Java is using our older Batch implementation. We had the same in Python but couldn't find anyone skilled in Java to migrate to the new Batch Report Item Failure mechanism (pure I/O vs AWS SDK manual deletes).

The only feature we might want to bring over is the ability to set a permanent exception to move an item to a DLQ (future, not now).

Shout if there's any other question you might have.

Thank you so much for helping out!!

[heitorlessa]

Here's the direct link where we based our implementation (similarly for SQS and Kinesis)

https://docs.aws.amazon.com/lambda/latest/dg/with-ddb.html

Section: Reporting batch item failures

There's an example for Python and Java. Here's one of our test events generated back then too

https://github.com/awslabs/aws-lambda-powertools-python/blob/develop/tests/events/dynamoStreamEvent.json

[driimus]

Have you experienced it otherwise when deployed?

Haven't had any issues, just wanted to double-check whether it's a possibility as I'd like to account for it in my internal type definitions.

For reference, I'm sourcing most types from @types/aws-lambda, where the aforementioned optional fields are declared optional, but batchItemFailures expects every itemIdentifier to be a string.

[heitorlessa]

Ah that's great, I thought something regressed.

I'd have to test in TS but we get that sequence number as a string - our data validation model below. Checked the official docs real quick, and both Java & Python examples uses the sequence number it receives it too (str).

Let me know if there's anything we can do to help

https://github.com/awslabs/aws-lambda-powertools-python/blob/develop/aws_lambda_powertools/utilities/parser/models/dynamodb.py