Bug: Payload Hash is Not Set on IdempotentRecord When Retrieving from Dynamo & Failing Validation

[brianhyder]

Expected Behaviour

When payload validation is enabled, the idempotency package should retrieve the item from dynamo and create an IdempotentRecord which includes the payload hash.

Current Behaviour

When payload validation is enabled, the idempotency package retrieves the item from dynamo but does not properly set the payload hash on the IdempotentRecord object. When the payload hash from the previous request is compared with the current payload hash, an error is thrown because the hash does not match the retrieved hash (set to undefined).

Code snippet

Configuration:

const config = new IdempotencyConfig({
   hashFunction: 'md5',
   useLocalCache: false,
   expiresAfterSeconds: 3600,
   throwOnNoIdempotencyKey: false,
   payloadValidationJmesPath: 'body',
   eventKeyJmesPath: 'headers.idempotency-key'
});
const persistenceStore = new DynamoDBPersistenceLayer({
   tableName: 'idempotency_store',
   keyAttr: 'key'
});
makeHandlerIdempotent({ config, persistenceStore });

Offending Code: DynamoDBPersistenceLayer->_getRecord()
https://github.com/awslabs/aws-lambda-powertools-typescript/blob/main/packages/idempotency/src/persistence/DynamoDBPersistenceLayer.ts#L108

Code that triggers error:
https://github.com/awslabs/aws-lambda-powertools-typescript/blob/main/packages/idempotency/src/persistence/BasePersistenceLayer.ts#L112

Steps to Reproduce

1. Use the configuration shown above
2. Execute a POST request against the configuration with an unique idempotency key
3. Verify that the dynmodb record was set and that the payload hash was set in on the item
4. Repeat the request with the same idempotency key and the same body
5. A validation error is thrown.

Possible Solution

Update the DynamoDBPersistenceLayer->_getRecord() to include the validation attribute value when creating the IdempotentRecord object.

return new IdempotencyRecord({
      idempotencyKey: item[this.keyAttr],
      status: item[this.statusAttr],
      expiryTimestamp: item[this.expiryAttr],
      inProgressExpiryTimestamp: item[this.inProgressExpiryAttr],
      responseData: item[this.dataAttr],
      payloadHash: item[this.validationKeyAttr]
    });

Also check to make sure all of the other possible properties are capable of being set.
NOTE: the above was not tested but based on line-by-line debugging.

Powertools for AWS Lambda (TypeScript) version

latest

AWS Lambda function runtime

18.x

Packaging format used

npm

Execution logs

Error: Payload does not match stored record for this event key
    at DynamoDBPersistenceLayer.validatePayload (/workspaces/lambda/service/node_modules/@aws-lambda-powertools/idempotency/src/persistence/BasePersistenceLayer.ts:306:15)
    at DynamoDBPersistenceLayer.getRecord (/workspaces/lambda/service/node_modules/@aws-lambda-powertools/idempotency/src/persistence/BasePersistenceLayer.ts:104:10)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at before (/workspaces/lambda/service/node_modules/@aws-lambda-powertools/idempotency/src/middleware/makeHandlerIdempotent.ts:83:11)
    at runMiddlewares (/workspaces/lambda/service/node_modules/@middy/core/index.cjs:159:21)
    at runRequest (/workspaces/lambda/service/node_modules/@middy/core/index.cjs:119:9)
    at Object.perform (/workspaces/lambda/service/src/utils/performerWrapper.js:28:22)
    at Object.<anonymous> (/workspaces/lambda/service/test/integration/createTransfer.spec.js:95:38) {stack: 'Error: Payload does not match stored record f…est/integration/createTransfer.spec.js:95:38)', message: 'Payload does not match stored record for this event key'}

[dreamorosi]

Hi @brianhyder thank you for reporting the bug. We'll take a look and get back to you in this same thread.

[dreamorosi]

Hi @brianhyder, I can confirm that I'm able to reproduce the issue and indeed there's a bug in the idempotency record creation upon retrieval.

I followed the call stack and like you pointed out, even when the validation field is saved in the DynamoDB table it's not being used when rehydrating the IdempotencyRecord.

I'm going to run a few more requests and check that no other fields is being dropped as well as adding tests to reduce the risk for future regressions on this topic. After that I'll push a PR.

Thanks for taking the time to not only report the issue but also investigate the issue, that was helpful.

[dreamorosi]

I found also another bug related to this feature: when saving or updating the record with validation enabled we were not using the JMESPath expression provided (the two linked lines should call this function which needs to be updated to match this implementation).

With this issue, even keeping in account the validation field discussed above, the payload wouldn't match. I'm fixing that too.

As a side note, assuming that your event looks like this:

{
  "headers": {
    "idempotency-key": "1234"
  }
}

then the JMESPath expression should be headers."idempotency-key" instead of headers.idempotency-key. According to the JMESPath spec, when the identifier contains special characters (like the hyphen in this case), it must be wrapped so that the lexer interprets it as a quoted identifier. You can verify that this is true using this online validator.

[dreamorosi]

The PR is up, if you have bandwidth feel free to give it a try - otherwise it'll be merged once @am29d reviews it.

[brianhyder]

No worries! I'm happy to help. I'll review the PR this morning. Thank you for the info on the JMES path.

[github-actions]

⚠️ COMMENT VISIBILITY WARNING ⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.