Feature request: improve error messaging around IdempotencyPersistenceLayerError

[cgoettel]

Use case

I was trying to implement the basic example using Terraform to deploy the Lambda and DynamoDB. However, I had forgotten to include the policy for the Lambda to talk to DynamoDB and kept getting the following error:

[ERROR] IdempotencyPersistenceLayerError: Failed to save in progress record to idempotency store
Traceback (most recent call last):
  File "/var/task/aws_lambda_powertools/middleware_factory/factory.py", line 134, in wrapper
    response = middleware()
  File "/var/task/aws_lambda_powertools/utilities/idempotency/idempotency.py", line 79, in idempotent
    return idempotency_handler.handle()
  File "/var/task/aws_lambda_powertools/utilities/idempotency/base.py", line 97, in handle
    return self._process_idempotency()
  File "/var/task/aws_lambda_powertools/utilities/idempotency/base.py", line 116, in _process_idempotency
    raise IdempotencyPersistenceLayerError("Failed to save in progress record to idempotency store") from exc

Each error message is the same which isn't entirely helpful. So I dug through the code and could see that it was exception chaining, but that also wasn't particularly helpful. It wasn't until I tried to implement the basic example using AWS SAM that I was able to see the policy that I'd forgotten to attach.

Solution/User Experience

Ideally, something in the error chain would receive a bad permissions error and that information would percolate up to the user. Every other time I deploy into AWS I get clear permissions errors and the reason I didn't even consider a permissions issue is because it was always so clear previously when that was it.

Alternative solutions

No response

Acknowledgment

• This feature request meets Lambda Powertools Tenets
• Should this be considered in other Lambda Powertools languages? i.e. Java, TypeScript

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #python channel on our AWS Lambda Powertools Discord: Invite link

[mploski]

Hey @cgoettel, thanks for raising this issue. I agree we need to enrich the error message here to be more meaningful. I will start working on this in a few coming days (probably end of the week or beginning of the next one). Will update you about the progress

[mploski]

Hey @cgoettel. I was able to reproduce mentioned issue and I can confirm your observation, that lambda emits logs in the shape you shared in the ticket itself. Let me share some details about it, just for future reference or interested readers :-)


In idempotency feature we rely on “raise from” statement to raise an exception with extra information whenever we have problem. That works if I trigger handler code directly from my local environment (not from lambda). Then we get chained traces like in the following snippet:



urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPSConnection object at 0x11324b520>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known

....


During handling of the above exception, another exception occurred:


raise EndpointConnectionError(endpoint_url=request.url, error=e)
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: 
...
The above exception was the direct cause of the following exception:

...
aws_lambda_powertools.utilities.idempotency.exceptions.IdempotencyPersistenceLayerError: Failed to save in progress record to idempotency store




Lambda runtime behaves differently though. In a runtime bootstrap action we modify logging behavior.


We do multiple different actions there, but for our case what's important is that exceptions not handled gracefully in handler itself are managed by another exception block. This block is responsible for extracting an exception and traceback, formatting message and sending it to a stdout. Specifically, we use sys.exc_info() function to extract raised exception. It also means we skip all other nested exceptions as they are not returned by exc_info() function. That's why we cannot see them in Cloudwatch logs.





If you are interested in details please take a look at a lambda runtime bootstrap code: https://github.com/aws/aws-lambda-python-runtime-interface-client/blob/970e9c1d2613e0ce9c388547c76ac30992ad0e96/awslambdaric/bootstrap.py#L157





I will work on exposing additional context from nested exceptions to make it visible in CloudWatch Logs.

[mploski]

hey @cgoettel, we just merged changes to the Idempotency feature, that will improve error messaging and hopefully will simplify debugging or reasoning about potential issues in future. Thank you one more time for the contribution and making Lambda Powertools better :-)

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.

[github-actions]

This is now released under 2.6.0 version!

[mattyweb]

Lifesaver! Thanks!

Just found this thread and upgraded to the new PowerTools in order to troubleshoot. Trying an idempotent lambda in python and this was my first time using DynamoDB.

Had to add the following actions to my lambda config:
"Action": [
"dynamodb:GetItem",
"dynamodb:PutItem",
"dynamodb:UpdateItem",
"dynamodb:DeleteItem"
],

Don't know how I would have figured this out otherwise. Might be good to add this to the PowerTools Idempotency docs! (Unless I just missed it!)

[heitorlessa]

We somehow missed it in the docs 😳! Apologies from the team. We always add within getting started—>required resources, with an IAM section when needed. For Idempotency docs, we added in the sample SAM template under Policies — however, we missed a dedicated section since we know not everyone uses SAM If someone wants to make their first PR, please feel free to do so and we can help with the wording - otherwise we’ll get it done this week. https://awslabs.github.io/aws-lambda-powertools-python/2.7.1/utilities/idempotency/#required-resources

…

On Mon, 6 Feb 2023 at 00:28, Matt Webster ***@***.***> wrote: Lifesaver! Thanks! Just found this thread and upgraded to the new PowerTools in order to troubleshoot. Trying an idempotent lambda in python and this was my first time using DynamoDB. Had to add the following actions to my lambda config: "Action": [ "dynamodb:GetItem", "dynamodb:PutItem", "dynamodb:UpdateItem", "dynamodb:DeleteItem" ], Don't know how I would have figured this out otherwise. Might be good to add this to the PowerTools Idempotency docs! (Unless I just missed it!) — Reply to this email directly, view it on GitHub <#1772 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZPQBFDYTWJ52USUML3LDLWWAZQPANCNFSM6AAAAAASZ2GYEE> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.*** com>

[heitorlessa]

@mattyweb we've just patched the live documentation with the IAM section for Idempotency - took us slightly longer to avoid leaking new features coming in the next release.

https://awslabs.github.io/aws-lambda-powertools-python/2.7.1/utilities/idempotency/#iam-permissions

Thank you again for flagging this!

[mattyweb]

Totally missed the update here. Thanks so much for the doc fix!