Credentials Provider bug with powertools-idempotency and Lambda SnapStart

[mhermus]

Hi! While using version 1.15.0 of powertools-idempotency and java11 Lambda runtime, I started using AWS SnapStart and my function broke (previously working fine). I suspect it is an issue with the credentials provider chain not using the container provider like the default chain is supposed to (which I understand is required to work with SnapStart). As a result, it currently seems impossible to use this library with SnapStart, as I cannot seem to override the credentials provider.

The error message below seems to clearly indicate this. NOTE: the exception looks like it went to System.out - in other words, it isn't a Log4J log message.

EDIT: I looked in the persistence code and found this, which seems to confirm my suspicions:

DynamoDbClientBuilder ddbBuilder = DynamoDbClient.builder()
                        .credentialsProvider(EnvironmentVariableCredentialsProvider.create())
                        .httpClient(UrlConnectionHttpClient.builder().build())
                        .region(Region.of(System.getenv(AWS_REGION_ENV)));
                this.dynamoDbClient = ddbBuilder.build();

in: DynamoDBPersistenceStore.java

Expected Behavior

The idempotency library works with AWS SnapStart as normal.

Current Behavior

The runtime cannot find credentials and therefore fails to write records to DynamoDB.

Possible Solution

Fix code to use default credentials provider chain that includes the container provider?

Steps to Reproduce (for bugs)

I suspect any SnapStart enabled Lambda will have this issue, but if not that would be great to know.

Environment

• Powertools version used: 1.15.0
• Packaging format (Layers, Maven/Gradle): Maven
• AWS Lambda function runtime: Java11
• Debugging logs

How to enable debug mode**

Failed to save in progress record to idempotency store. If you believe this is a powertools bug, please open an issue.: software.amazon.lambda.powertools.idempotency.exceptions.IdempotencyPersistenceLayerException
software.amazon.lambda.powertools.idempotency.exceptions.IdempotencyPersistenceLayerException: Failed to save in progress record to idempotency store. If you believe this is a powertools bug, please open an issue.
	at software.amazon.lambda.powertools.idempotency.internal.IdempotencyHandler.processIdempotency(IdempotencyHandler.java:91)
	at software.amazon.lambda.powertools.idempotency.internal.IdempotencyHandler.handle(IdempotencyHandler.java:66)
	at software.amazon.lambda.powertools.idempotency.internal.IdempotentAspect.around(IdempotentAspect.java:74)
	at com.mindcareone.auth.handler.PostConfirmationHandler.handleRequest(PostConfirmationHandler.java:81)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId).
	at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
	at software.amazon.awssdk.auth.credentials.internal.SystemSettingsCredentialsProvider.resolveCredentials(SystemSettingsCredentialsProvider.java:58)
	at software.amazon.awssdk.core.internal.util.MetricUtils.measureDuration(MetricUtils.java:50)
	at software.amazon.awssdk.awscore.internal.authcontext.AwsCredentialsAuthorizationStrategy.resolveCredentials(AwsCredentialsAuthorizationStrategy.java:100)
	at software.amazon.awssdk.awscore.internal.authcontext.AwsCredentialsAuthorizationStrategy.addCredentialsToExecutionAttributes(AwsCredentialsAuthorizationStrategy.java:77)
	at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.invokeInterceptorsAndCreateExecutionContext(AwsExecutionContextBuilder.java:120)
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.invokeInterceptorsAndCreateExecutionContext(AwsSyncClientHandler.java:69)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:78)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:179)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76)
	at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56)
	at software.amazon.awssdk.services.dynamodb.DefaultDynamoDbClient.putItem(DefaultDynamoDbClient.java:4243)
	at software.amazon.lambda.powertools.idempotency.persistence.DynamoDBPersistenceStore.putRecord(DynamoDBPersistenceStore.java:148)
	at software.amazon.lambda.powertools.idempotency.persistence.BasePersistenceStore.saveInProgress(BasePersistenceStore.java:162)
	at software.amazon.lambda.powertools.idempotency.internal.IdempotencyHandler.processIdempotency(IdempotencyHandler.java:84)
	... 7 more

[mhermus]

Just looked at the code, and confirmed issue in: powertools-idempotency/src/main/java/software/amazon/lambda/powertools/idempotency/persistence/DynamoDBPersistenceStore.java

The code is explicitly using the environment credentials provider.

[msailes]

Thanks for reporting this. Before the fix is released you can create your own DynamoDbClient and pass it in to the withDynamoDbClient method of DynamoDBPersistenceStore.

For example:

Idempotency.config().withConfig(
                IdempotencyConfig.builder()
                        .withEventKeyJMESPath("powertools_json(body).address")
                        .build())
        .withPersistenceStore(
                DynamoDBPersistenceStore.builder()
                        .withDynamoDbClient(client)
                        .withTableName(System.getenv("IDEMPOTENCY_TABLE"))
                        .build()
        ).configure();

[msailes]

@mhermus, it would be good to chat about the fix if possible msailes at amazon dot co dot uk

[mhermus]

Brilliant, thanks! I didn't catch that option.

[jeromevdl]

fixed in #1161

[scottgerring]

Released in 1.16.0