Manipulation of Idempotent response

[royassis]

Use case

We have a client-server web application.
The server side is built using AWS SAM with an API Gateway and multiple lambda functions. The server side laso executes an asynchronous pipeline.

We want to prevent cases where a duplicated message is sent to the backend and triggers the pipeline multiple time.
We also want to notify the user if a duplication occurred, and hence his request was not processed, this is the main issue.

Solution/User Experience

Add an X-Idempotent: True header to the response.

Alternative solutions

A hook for changing the response (e.g. adding a field).

Acknowledgment

• This feature request meets Lambda Powertools Tenets
• Should this be considered in other Lambda Powertools languages? i.e. Java, TypeScript

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #python channel on our AWS Lambda Powertools Discord: Invite link

[heitorlessa]

cc @walmsles as I'm sure you'll have ideas on DX here ;)

[leandrodamascena]

Original discussion: #2158

[walmsles]

Interesting - always interesting to hear more about different solution needs. The complexity around this is also that the idempotent handler runs as a Python decorator either on the lambda_handler or other function in the call chain and has zero knowledge of where it fits or where the event came from, which should be maintained.

We must also be clear that Idempotent responses are not just about API responses, so feel this is not a standard library feature that Powertools should provide "out of the box" in a canned way, as indicated by the initial request.

I think a mechanism for "intercepting" the idempotent response is useful and would provide @royassis with the capability to implement the customisation for his use case. This also provides future customisation opportunities for anyone. How the response is sent back is entirely up to the developer within the context of their solution, which is where control needs to be placed for any customisation of an idempotent response (in developer-maintained code).

Here is an example of what I am thinking using Powertools API Gateway resolver and an Idempotent Intercept at the API method level, which makes sense for an API use case (Note: I know an idempotent decorator for API methods is not really a thing today - but it should be and would be perfect for this use):

import json
from http import HTTPStatus

import requests

from aws_lambda_powertools import Logger, Tracer
from aws_lambda_powertools.event_handler import (
    APIGatewayRestResolver,
    idempotent_api,
    Response,
    content_types,
)
from aws_lambda_powertools.utilities.idempotency import (
    DynamoDBPersistenceLayer, IdempotencyConfig
)
from aws_lambda_powertools.logging import correlation_paths
from aws_lambda_powertools.shared.cookies import Cookie
from aws_lambda_powertools.utilities.typing import LambdaContext

tracer = Tracer()
logger = Logger()
app = APIGatewayRestResolver()

persistence_layer = DynamoDBPersistenceLayer(table_name="IdempotencyTable")

def payment_intercept(payload : dict, idempotent_response: bool):
    response = Response(
        status_code=HTTP.OK.value,
        body=payload,
        content_type=content_types.APPLICATION_JSON,
        header={"X-Idempotent": idempotent_response}
    )

    return response

@app.post("/pay")
@idempotent_api(resolver=app, event_key_jmespath="powertools_json(body).[user, product_id]", persistence_store=persistence_layer, response_intercept=payment_intercept)
@tracer.capture_method
def process_payment():
    payment = create_subscription_payment(
        user=app.current_event.json_body['user'],
        product=app.current_event.json_body['product_id']
    )
    ...
    return Response(
        status_code=HTTP.OK.value,
        body=json.dumps(
            {
                "payment_id": payment.id,
                "message": "success",
            }
        ),
        content_type=content_types.APPLICATION_JSON
    )


@logger.inject_lambda_context(correlation_id_path=correlation_paths.API_GATEWAY_REST)
@tracer.capture_lambda_handler
def lambda_handler(event: dict, context: LambdaContext) -> dict:
    return app.resolve(event, context)

The above implies:

• idempotent_api() decorator is created and works (just thinking about it right now) - I see this as a useful thing and applying idempotency at the API resolution layer is easy to reason about if you have a monolithic API
• idempotency interceptor - would require an internal decorator to intercept idempotency response and a change to idempotency internally to return the payload AND whether the result was read from the Cache. This shouldn't break anything as the idempotency internal interceptor would hide this fact from existing use-cases.
• idempotency itself may need a newer class to unwrap API Response Objects for storing in the PersistenceProvider.

[heitorlessa]

got some security enhancements I need to prioritize first, but adding this to be looked after in the next iteration (next week).

Thank you everyone!!

[heitorlessa]

Haven't forgotten, our security enhancements are taking longer than I expected (to give you a glimpse of the complexity), plus COVID took a hit on me (back now 🙌).

Adding help wanted as we'd appreciate anyone taking a stab at this - @royassis feel free to give it a try and we can help refine it.

---

What we seem to be after:

• A mechanism to call an user-defined function for requests that are idempotent (think Cache: Hit -> hook)
• Optionally a new route parameter or decorator to more easily make a route idempotent and include a header as originally asked

I say, optionally, because it's only a matter of time until someone needs something else, and eventually it'll bloat in complexity and cognitive load to remember many parameters.

A hook-like would be useful to give full flexibility for customers to do as they please, and it'd only be called IF the transaction is idempotent - making it suitable for all sorts of use cases.

What we don't want

We should not modify the idempotent response of a decorated method. I think this is a better job for a deserializer (cc @ran-isenberg) in case you want to deserialize the JSON serialized idempotent response into something that better match your method return signature (e.g., Idempotency, Dataclass, Custom, etc.). Happy to hear counter arguments and learn from it too, specially how tricky it'd be from the typing system.

[ran-isenberg]

@heitorlessa I was thinking of adding a JSON serializer/deserializer optional arguments to the config class. That way I would have total control of the way it is saved to the database and loaded back from. What do you think?

[heitorlessa]

@heitorlessa I was thinking of adding a JSON serializer/deserializer optional arguments to the config class. That way I would have total control of the way it is saved to the database and loaded back from. What do you think?

sorry I missed... I could definitely see a deserializer but unsure on serializer as it's easy to shoot yourself in the foot e.g., fails to store in DB because it doesn't match the storage requirements.

Feel free to open up a feature request :)

[Shadow798]

Hi,

just to add some other perspective here. I also have very similar use case. For my case I need to know whether response returned by my function decorated with idempotent_function decorator was original one or the one fetched from dynamo - essentially I need to know whether message was already previously processed or not. If it was then I don't want to run my main handler code at all, because I'm using messages to increment some values. If idempotent function returns the message saved in Dynamo my code will still use this message to increment the value causing end data to be wrong. So I want to have the distinction and be able to simply filter out if message was already processed.

I'm using idempotency with batch integration.

Right now I did some "dirty" workaround and modified the response of decorator to also include flag that tells if message was returned from Dynamo or not (if yes then I know it was already processed). I'm capturing this flag in _process_idempotency method (specifically in except IdempotencyItemAlreadyExistsError block). So in batch response my function response is a tuple, first element contains actual message, second if it was already processed or not. Not the best from design perspective, but works for my use case.

Personally simple flag indicating this in the response would be enough to solve my problem. Additional point that comes to my mind would be if we want to have a counter that will tell how many times message was returned from storage. Not sure if there would be any use of this, but wanted to give a heads up on this idea.

Thanks
Karol

[heitorlessa]

Thanks a lot @sthulb for looking into it this month!

[heitorlessa]

resurfacing this potentially to the next iteration cycle -- first order is to get the new Parameters/Secrets Put and AppSync Batch invoke long-standing ones.

[TonySherman]

Would like to add my use for this feature in case it helps with the development of this.

I have a statemachine that invokes a lambda that does a lookup and in the next step, it sends an email with SES. In order to utilize the idempotency utility, I would need some way to modify the lambda response, so I could add a choice step in my statemachine to decide to send the email or end the execution.

Something like the proposed idea above that can intercept the response and have access to the Record status would work great. @idempotent(persistence_store=persistence_layer, ... response_intercept=payment_intercept)

Then I can just look for the Record status in the response from my lambda to decide whether to continue execution or end.

[walmsles]

Just want to mention that there are 2 parts to this issue from the discussions.

1. Allow for the manipulation of idempotent response when it is returned (for idempotent middleware over handler)
2. Integrate item 1 better with EventHandler use-cases for simpler handling of idempotency per route if desired.

The second part - I am considering the best way to integrate and will do a POC in the coming days.

[leandrodamascena]

Hello everyone! I'm updating this issue with some decisions we made regard the Developer Experience and request flow.

Response hooks

We are introducing support for hooks in the Idempotency utility. Customers can now create a function and pass it as the hook in the IdempotencyConfig. This hook will be invoked every time a request is idempotent. If the request is not idempotent, the hook will not execute.

from aws_lambda_powertools.utilities.idempotency import IdempotencyConfig
def my_response_hook(response: Dict, idempotent_data: DataRecord): ...
config = IdempotencyConfig(response_hook=my_response_hook)

Manipulating response

This hook will be able to access both: the response from the invocation and the DataRecord from the Idempotency Store. This means customers can inject new keys into the response or utilize attributes from the DataRecord to log information or create metrics, for example.

Persistent Store

This hook is compatible with any Persistent Store. Currently, we support DynamoDB and Redis for this implementation.

Handling exceptions

Customers are responsible for handling exceptions when using Response hooks. During the implementation of this new feature, we considered the possibility of implementing soft failure or raising warnings to ignore hook execution. For instance, this approach could prevent customers from having exceptions in the hook, thereby avoiding potential malfunctions in the Lambda function execution. However, we opted against this approach because it could result in unexpected behaviors and potentially breaking the contract that customers expect for Response hooks. Let me provide some context with an example.
Consider a scenario where a customer creates a hook to inject a specific key into the response when the operation is idempotent and wants to use it to trigger an alert for third-party integration. The code might appear like this:

def my_response_hook(response: Dict, idempotent_data: DataRecord) -> Dict:
    # Return inserted Header data into the Idempotent Response
    response["x-idempotent-key"] = idempotent_data.idempotency_key

    imagine_an_exception_here()

    # expiry_timestamp could be None so include if set
    expiry_timestamp = idempotent_data.expiry_timestamp
    if expiry_timestamp:
        expiry_time = datetime.datetime.fromtimestamp(int(expiry_timestamp))
        response["x-idempotent-expiration"] = expiry_time.isoformat()

    # Must return the response here
    return response

dynamodb = DynamoDBPersistenceLayer(table_name="IdempotencyTable")
config = IdempotencyConfig(response_hook=my_response_hook)

@idempotent_function(data_keyword_argument="order", config=config, persistence_store=dynamodb)
def process_order(order: dict) -> dict: ...

def lambda_handler(event: dict, context: LambdaContext):
    config.register_lambda_context(context)  # see Lambda timeouts section
    process_order = process_order(event.get("order"))

    if process_order["x-idempotent-key"] is not None:
        trigger_some_sns_alert()

If we choose to ignore the hook execution exception and only raise a warning instead of propagating the exception, customers may encounter difficulties in debugging. They may struggle to understand why the if process_order["x-idempotent-key"] is not None: condition is failing and why the x-idempotent-key doesn't exist in the response.

Documentation

The documentation will be updated to inform customers on the best practices for using Response hooks. Additionally, it will feature an example demonstrating how to utilize Response hooks and effectively handle exceptions.

We welcome any suggestions or ideas for improvements regarding this matter.

Thank you to everyone who contributed their ideas.

[leandrodamascena]

Just want to mention that there are 2 parts to this issue from the discussions.

1. Allow for the manipulation of idempotent response when it is returned (for idempotent middleware over handler)
2. Integrate item 1 better with EventHandler use-cases for simpler handling of idempotency per route if desired.

The second part - I am considering the best way to integrate and will do a POC in the coming days.

Hi @walmsles! After we merge PR #4037, I will open another issue to implement the Response hooks in the Event Handler.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.