Bug: in_progress_expiration field is not set in Idempotency record when too close to lambda timeout

[bertpl]

Expected Behaviour

In case config.register_lambda_context(...) is called correctly before calling an idempotent_function-decorated function, the record saved to the persistence layer should always contain an in_progress_expiration field

Current Behaviour

It is an observed behavior of AWS that it kills lambda functions slightly after their timeout has been reached, e.g. after 300.68 seconds, if the lambda timeout is set to 300s.

This creates a time window of 0.68s long within which lambda_context.get_remaining_time_in_millis() returns 0 (the true value would be negative).

However, this situation is misinterpreted by the DynamoDBPersistenceLayer as this value actually being None, which corresponds with the case of config.register_lambda_context(...) not having been called correctly. As a result, the in_progress_expiration field of the idempotency record is not set correctly.

If then the lambda times out before exiting the decorator function, the record will never bet set to status COMPLETED, nor will it ever expire (before the actual TTL expiration, which typically is much larger). This triggers an infinite loop of retries, each time encountering a IdempotencyAlreadyInProgressError, similar to what is described here: #1038.

Code snippet

config = IdempotencyConfig(expires_after_seconds=24*60*60)

persistence_store = DynamoDBPersistenceLayer(table_name="my-idempotency-table")

@idempotent_function(
    data_keyword_argument="task",
    persistence_store=persistence_store,
    config=config,
)
def process_my_task(task: dict) -> None:
    ...


if __name__=="__main__":

    # create a dummy lambda context that reproduces a situation where we are very close
    # (or slightly beyond) the lambda timeout
    dummy_lambda_context = LambdaContext(
        invoke_id=str(uuid4()),
        client_context=dict(),
        cognito_identity=dict(),
        epoch_deadline_time_in_ms=int(1000 * time.time())  # fakes a lambda timeout = now
    )

    # call register_lambda_context as if we're in a lambda handler
    config.register_lambda_context(dummy_lambda_context)

    # call function -> this will trigger creation of an idempotency record without
    #                  a 'in_progress_expiration' field 
    process_my_task(task=dict(do="some_stuff"))

Possible Solution

The bug seems to boil down to an improper check inside the BasePersistenceLayer class (base.py, line 304 in my version of the package):

The check if remaining_time_in_millis results in False both in case of None and 0, while it actually should only result in False when the value is None.

So my suggestion would be to replace this check with if remaining_time_in_millis is not None:. This seemingly would solve the problem.

Steps to Reproduce

See code snippet.

Powertools for AWS Lambda (Python) version

2.38.1

AWS Lambda function runtime

3.11

Packaging format used

PyPi

Debugging logs

No response

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

[sthulb]

Thanks for the report @bertpl – we'll create a fix for this before the next release

[leandrodamascena]

Hey @bertpl! This is the kind of edge case that's hard to think through and figure out how to solve. Thanks a lot for taking the time to reproduce the error and sending us a snippet that made testing so much easier for us. 🚀

[leandrodamascena]

Closed via #4773

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.

[github-actions]

This is now released under 2.42.0 version!

[bertpl]

Thx @leandrodamascena & @dreamorosi for fixing this so quickly! Happy to have been able to contribute :-)

[leandrodamascena]

Thx @leandrodamascena & @dreamorosi for fixing this so quickly! Happy to have been able to contribute :-)

We mentioned your contribution in our release notes: https://github.com/aws-powertools/powertools-lambda-python/releases/tag/v2.42.0.

Thanks again 🚀