Feature request: set correlation ID in Logger

[dreamorosi]

Use case

When working with logs in distributed systems like AWS Lambda, customers want to be able to set arbitrary correlation IDs for their workloads.

Today this is possible using existing Logger features (see "Alternative solutions" section), however the experience is somewhat verbose and could be improved.

Powertools for AWS Lambda (Python) has a feature that allows customers to include a correlation_id key to each log. This feature request proposes to implement the same feature in this version of Powertools for AWS.

This would be a first, low hanging, effort for allowing customers to more easily work with correlation IDs with Powertools for AWS (see #129).

Important

If you're interested in this feature, please consider leaving a 👍 under this post, or leave a comment - measuring demand and having customer names helps us prioritize features in our backlog.

Solution/User Experience

The new feature is primarily aimed at customers using the injectLambdaContext class method decorator and Middy.js middleware.

Given an event with this type:

type MyEvent = {
  headers: {
    my_request_id_header: string
  }
}

When using decorators, the experience would look like this:

import { Logger } from '@aws-lambda-powertools/logger';
import type { Context } from 'aws-lambda';
import type { LambdaInterface } from '@aws-lambda-powertools/commons/types';

const logger = new Logger();

class Lambda extends LambdaInterface {
  @logger.injectLambdaContext({ correlationIdPath: 'headers.my_request_id_header' })
  public async handler(event: MyEvent, context: Context) {
    // ...
  }
}

When using the Middy.js middleware, the experience would instead look like this:

import { Logger } from '@aws-lambda-powertools/logger';
import { injectLambdaContext } from '@aws-lambda-powertools/logger/middleware';
import middy from '@middy/core';

const logger = new Logger();

export const handler = middy(async (event: MyEvent) => {
  // ...
}).use(injectLambdaContext(logger, {
  correlationIdPath: 'headers.my_request_id_header'
});

For the manual instrumentation the method is less compelling, but also a very low hanging fruit as it would just be an alias for logger.appendKeys({ correlation_id: ... }) (see docs):

import { Logger } from '@aws-lambda-powertools/logger';

const logger = new Logger();

export const handler = async (event: MyEvent) => {
  logger.setCorrelationId(event.headers?.headers.my_request_id_heeader);
  
  // ...
}

Each one of these snippets would generate a log entry similar to this:

{
    "level": "INFO",
    "message": "Collecting payment",
    "timestamp": "2021-05-03 11:47:12,494+0000",
    "service": "payment",
    "cold_start": true,
    "function_name": "test",
    "function_memory_size": 128,
    "function_arn": "arn:aws:lambda:eu-west-1:12345678910:function:test",
    "function_request_id": "52fdfc07-2182-154f-163f-5f0f9a621d72",
    "correlation_id": "correlation_id_value" // <- here
}

The feature would also come with a handful built-in correlation ID expressions, for example:

• API_GATEWAY_REST => "requestContext.requestId"
• APPSYNC_RESOLVER => 'request.headers."x-amzn-trace-id"'
• full list here

These would be used like this:

import { Logger } from '@aws-lambda-powertools/logger';
import { API_GATEWAY_REST } from '@aws-lambda-powertools/logger/correlation-paths';
import type { Context } from 'aws-lambda';
import type { LambdaInterface } from '@aws-lambda-powertools/commons/types';

const logger = new Logger();

class Lambda extends LambdaInterface {
  @logger.injectLambdaContext({ correlationIdPath: API_GATEWAY_REST })
  public async handler(event: MyEvent, context: Context) {
    // ...
  }
}

Open questions

Note

The points below should be addressed before the feature is ready to be implemented, if you want to contribute please help us instead of opening a PR.

The current implementation in Powertools for AWS Lambda (Python) relies on JMESPath expressions for extracting the correlation ID from the payload. This choice was sensible for them both because of their ecosystem but also because they already have the jmespath module in their dependency tree because of other features.

For us this is still a decision point to be addressed. Taking a dependency on @aws-lambda-powertools/jmespath that we released for the Idempotency utility would be easy in terms of complexity, however before moving forward with this I’d like us to explore 1/ whether it’s possible to set the dependency as optional peerDependency and load it only when using the feature, and if not, 2/ whether the impact on the bundle is acceptable.

I am not particularly inclined towards other JSON-query languages at this stage, but I am open to consider proposals if any. However implementing a function-based query feature (i.e. correlationId: (event) => (event.headers.my_request_id_header)) is out of question because of the reasons described in #1644 (unless anyone has compelling arguments).

Alternative solutions

import { Logger } from '@aws-lambda-powertools/logger';

const logger = new Logger();

export const handler = async (event: MyEvent) => {
  logger.setCorrelationId(event.headers?.headers.my_request_id_heeader);
  
  // ...
}

Acknowledgment

• This feature request meets Powertools for AWS Lambda (TypeScript) Tenets
• Should this be considered in other Powertools for AWS Lambda languages? i.e. Python, Java, and .NET

Future readers

Please react with 👍 and your use case to help us understand customer demand.

[thiagomeireless]

I'm sad to see this feature having only 1 👍🏻

I'm sure this is something multiple people would (should?) be using, especially in event source contexts. We have a lot of services that works like API Gateway entry point -> Lambda -> SNS/SQS -> Lambda and it's such a mess to debug things without setting up all the correlationId context, getters/setters, etc.

It would be great to have this functionality baked in PowerTools so we could just do correlationIdPath: 'headers.my_request_id_header'.

Hopefully this gets more traction so we can get feature parity with the Python version 🙏🏻

[dreamorosi]

Hi @thiagomeireless - thank you for the interest in the feature.

My main concern in adding the feature is having an extra dependency to Logger, which is widely used.

I'd like to find a way to make it optional unless you're using the feature, but I don't know how yet.

[dreamorosi]

I made a quick PoC of how this could look, and the DX would look something like this:

import { Logger } from '@aws-lambda-powertools/logger';
import { search } from '@aws-lambda-powertools/logger/correlationId';

const logger = new Logger({
  logLevel: 'debug',
  correlationIdSearchFn: search,
});

Behind the scenes the @aws-lambda-powertools/logger/correlationId is a re-export that looks something like this:

import { search as JMESPathSearch } from '@aws-lambda-powertools/jmespath';
import { PowertoolsFunctions } from '@aws-lambda-powertools/jmespath/functions';

const search = (expression, data) => JMESPathSearch(expression, data, { customFunctions: new PowertoolsFunctions() })

export { search }

Then later on customers could specify a correlationIdPath in the class method decorator or Middy.js middleware, which under the hood would use the correlationIdSearchFn passed in the constructor to extract the actual correlation ID and add it as a key to the logger.

By passing the reference to a JMESPath search function and exposing the JMESPath search function under a sub-path export, we can make the search function optional, and the JMESPath package a peer dependency so that customers who are not using the feature don't need to pay the overhead for it.

Implementation details

1. The @aws-lambda-powertools/jmespath package becomes a peer dependency of Logger
2. The ConstructorOptions type here would be extended with a new option:

{
  correlationIdSearchFn?: (expression: string, data: JSONObject) => unknown;
}

3. The setOptions() internal method here in the logger would store a reference to correlationIdSearchFn as private property
4. The InjectLambdaContextOptions type here would be extended to accept a new optional parameter called correlationIdPath (string) which opts-in into the feature
5. The injectLambdaContext class method decorator around here would use the this.#correlationIdSearchFn to extract a correlation id and add it as temporary attribute to the logger when the aforementioned `` is present. If the function is undefined it'd log a warning.
6. Same thing as point 4 but for the middleware

[am29d]

I ran some tests to understand the impact of the jmespath dependency for the logger, and the results were around 70kb of added code that customers won't use, if they don't use this feature.

I arrived at the same conclusion to have a wrapper around the search function and pass the instance via constructor, something we often do, i.e. ajv in validator or sdk client in other utilities.

The only disadvantage I see is that we would replicate the search signature in the logger, or other functions that we would add over time, i.e. envelopes. But I don't see the changes happening often, thus it's acceptable to maintain it in two places. I will now move ahead and submit a PR for review.

[github-actions]

⚠️ COMMENT VISIBILITY WARNING ⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.

[github-actions]

This is now released under v2.17.0 version!