Feature request: add support for ReturnValuesOnConditionCheckFailure in Idempotency

[dreamorosi]

Is your feature request related to a problem? Please describe.

When working with the Idempotency utility, the current implementation performs 2 operations: one write and one read whenever it encounters an idempotent request after the first time.

This happens because the implementation hinges on making an optimistic write to set the request as "in progress" while using a condition expression. When there's already another item, the condition expression will fail and the error will be interpreted as a request that should not be retried.

When this happens, the utility then goes on by reading the existing previously inserted error. The entire flow I just described is represented here in this diagram.

Several months ago DynamoDB announced the ability to specify a ReturnValuesOnConditionCheckFailure parameter that allows the call to automatically return the item that made the write fail

Implementing this mechanism would allow the utility to bypass the second read and thus improve performance and materialize cost savings for end customers.

Describe the solution you'd like

Currently the feature is supported in both the TypeScript and Python versions of Powertools for AWS. Here's the implementation we have in TypeScript for example:

• here is where we use the new flag
• here is where we handle the conditional check failed error, get the item, and pass it to the IdempotencyItemAlreadyExistsError we re throw
• here is where we conditionally use that idempotency record if present, or default to the old behavior of reading it in a separate operation

We could add something similar here.

Describe alternatives you've considered

N/A

Additional context

N/A

[phipag]

Hey @dreamorosi,

I just looked at this issue in more detail and will update some code to avoid making a second call to DynamoDB if the IdempotencyItemAlreadyExistsException is raised.

As a side note: Today, the Java runtime already avoids making a second to the DDB if the in-memory cache is enabled. Let me explain this process in more detail:

1. here is the entrypoint which is called by the Idempotency handler. This is the same saveInProgress method as in TypeScript.
2. If an IdempotencyItemAlreadyExistsException is thrown it will try to get the idempotency record in order to return the response payload. This is where a second call to DynamoDB is necessary if the in-memory cache is disabled and ReturnValuesOnConditionCheckFailure is not used.
3. In case in-memory cache is enabled, the cached DataRecord will be returned here. Otherwise, a second call to DynamoDB will be made to read the idempotency record.

Otherwise, a second call to DynamoDB will be made to read the idempotency record.

This is where we can improve by enabling ReturnValuesOnConditionCheckFailure. We can directly attach the idempotency DataRecord to the IdempotencyItemAlreadyExistsException exception and avoid calling the getIdempotencyRecord() method completely if a DataRecord already exists on the exception which is more cost-efficient (update this line to check the exception first.).

To summarize, the behavior will be:

1. New feature: Check IdempotencyItemAlreadyExistsException if a DataRecord was returned through ReturnValuesOnConditionCheckFailure.
2. If no, check in-memory cache (if enabled)
3. If not cached, fall back to making a DDB call and handle possible error scenario

I am working on a PR for this now.

[dreamorosi]

Thanks for the explanation.

In memory cache is good, but there's no guarantee that requests are served by the same Lambda execution environment, so it's still a good idea to add this.

Looking forward to see the PR!

[phipag]

I agree, and the in-memory cache is also optional.

I created a PR for this feature here: #1821

I would love your review when you can find some time.

[phipag]

Resolved by PR #1821