Idempotency and Error Handling

[brianhyder]

Hey, all.

As of the 1.10 build, all of our idempotency use cases are passing. A huge THANK YOU for getting this functionality squared away.

For this discussion, I wanted to walk through a couple of items to get your feedback. We had 2 use cases that the out-of-the-box makeHandlerIdempotent function did not cover:

1. When an subsequent API request is made with the same idempotent key but a different request body, a 409 status code should be returned.
2. All responses, including error responses, must be persisted
3. All error responses must be formatted as RFC7807 Problem documents

Our error handling philosophy is to create custom errors for different situations: 400 validation, 500 server/unhandled, etc. Those errors have properties attached to them so that our generic error handling middleware can properly pick them up and format them.

To satisfy requirement 2, we wrapped the makeHandlerIdempotent and the onError portion of the middleware. The default behavior is to delete any idempotent record on error. To ensure that the error response gets persisted, we needed the same behavior as if there was a successful response.

To satisfy requirement 1, we needed to capture the specific error that was thrown. You all made that easy as you created a custom error that is thrown in this situation. We simply added code to the onError handler we used to override the default behavior. This was the only case where we did not want to persist the outcome as it would override the values of the first request that was used with that idempotent key.

To satisfy requirement 3, we simply added the additional metadata we needed so that our generic error handler would set the proper response. This also meant that we needed to inject our generic error handler so that the error could be handled. As outlined below, the positioning of the error handlers in the middleware chain matters.

Putting it all together our implementation looks a bit like this:

lambda handler initialization:

const middlewares = [
  makeHandlerIdempotent({
    persistenceStore: {
      awsSdkV3Client: dynamoDBClient,
      tableName: process.env.LOCATION_OF_PERSISTENCE_STORE
    },
    errorHandler: errorHandler().onError
  }),
  otherMiddleware1(),
  otherMiddleware2(),
  errorHandler()
];

module.exports = {
  handler: middy(handler).use(middlewares)
};

Wrapped makeHandlerIdempotent function:

const instance = middleware.makeHandlerIdempotent({ config, persistenceStore });
  return {
    ...instance,
    /**
     * We override this error handler because of the way we bubble up our errors.  We throw on
     * every type of non-success status code.  AWS's implementation for idempotency deletes the
     * idempotency record if any error is thrown inside of the pipeline.  In our case, we want to
     * persist the record anyway regardless of error type.
     * @param {{error:Error}} request
     * @returns {Promise<void>}
     */
    onError: async (request) => {
      // in the case where the hash of the request body of the subsequent request does not match
      // the hash from the first request, an error is thrown.  This prevents general bad behavior.
      // The caller should NOT change anything about a retry.
      if (request.error && request.error instanceof IdempotencyValidationError) {
        request.error.status = 409;
        request.error.type = `${urlToDocs}#${request.error.status}`;
        request.error.title = 'Idempotency validation error';
        request.error.detail = request.error.message;
        if (options.errorHandler) {
          // when supplied, overwrite the response based on the modifications to the error
          await options.errorHandler(request);
        }
        // this is the only scenario when we don't want to re-save the idempotency record.
        return;
      }
      // now that errors are handled, call the default after handler for the idempotency module
      // which will persist the error result
      await instance.after(request);
    }

It is important to note that the position of the error handler matters.

• Error thrown in idempotency layer (validation error): the error handler must be in the first position with no other error handlers active
• Error thrown in lambda implementation: the error handler must be after the idempotency middleware
• Error thrown in other middleware (otherMiddleware1): the error handler must be after the idempotency handler

To mitigate each of these cases, we intentionally did 3 things:

1. The idempotency middleware was first in the chain
2. The generic error handler was last in the middleware chain
3. An instance of the error handler was injected into the makeHandlerIdempotent function as configuration so that it could handle the case where a subsequent request with a different body errors with a properly formatted response.

Anyway, feel free to send any feedback our way. Thank you for getting this built and ready. We'll be on the lookout for the first official release.

[dreamorosi]

Thank you for taking the time to write your experience with the utility and share it with the rest of the community.

With your permission we'll reuse/reword part of your solution for a new section dedicated to Idempotency that we will add to the Middy docs.