Feature request: Allow changing log level at runtime

[Aaronius]

Use case

If I have an issue in production, I want to set the debug log level to DEBUG right away. The fastest way for me to do that is through the AWS console. However, if I set the environment variable in the AWS console, the new log level doesn't seem to get picked up in existing execution environments, so the logger doesn't log any subsequent debug logs.

Solution/User Experience

I'm not sure if changing the environment variable in the AWS console updates environment variables within existing execution environments. If it does, then perhaps the logger could poll the environment variable every minute or so to make sure it's logging at the most recently configured level. Maybe there's a better way to handle this that I'm not considering.

Alternative solutions

No response

Acknowledgment

• This feature request meets Lambda Powertools Tenets
• Should this be considered in other Lambda Powertools languages? i.e. Python, Java

[saragerion]

However, if I set the environment variable in the AWS console, the new log level doesn't seem to get picked up in existing execution environments, so the logger doesn't log any subsequent debug logs.

This is not the expected behaviour and I'd like us to investigate about that's happening here. Can you please share with us the code you use to reproduce this issue?

[saragerion]

Note: potentially a bug, needs investigation.

[Aaronius]

Nothing special:

export default new PowertoolsLogger({
  serviceName: 'ciam',
  sampleRateValue: 0.3,
});

and then calling logger.debug with messages throughout our app.

FWIW, I'm not really sure how/when AWS gets updated environment variables into the runtime. If changing the log level at runtime is intended to be working as it is, I can play around with it more and see if I find anything new.

[am29d]

I tried to reproduce the issue with a simple sam app template:

• create sam-app with powertools ts (log level set to INFO)
• invoke the http API
• the log output in CW is INFO, log level set to INFO
• change the env var in lambda console LOG_LEVEL to DEBUG
• invoke the http API
• the log output in CW is INFO and DEBUG, log level set to DEBUG.

It seems like you create and export your own wrapper around the logger, can you provide more details how you define and use it?

[dreamorosi]

However, if I set the environment variable in the AWS console, the new log level doesn't seem to get picked up in existing execution environments

This is expected and is independent from Powertools. In AWS Lambda, environment variables are part of the function configuration and are locked for a specific version of a function.

To quote the docs section about Environment Variable in Lambda:

You define environment variables on the unpublished version of your function. When you publish a version, the environment variables are locked for that version along with other version-specific configuration.

Updating environment variables requires a call to the UpdateFunctionConfiguration API. When you update the variable via the AWS Console this is done behind the scenes for you.

Any change to the environment variables (or to the function's configuration / code) will always apply to the next execution environment spun by the service.

If you have an execution environment that's still active from before you updated the environment variables (or function's configuration / code), for a certain amount of time some requests might still be accepted by that old environment (which has now outdated configuration). The amount of time needed to drain the older execution environments can vary based on the workloads request volume, but this is completely opaque to the user, meaning you don't have control over this.

The only way you have to force discarding all existing execution environment for a given function is to throttle the function (aka set concurrency to 0), however this is considered a break-glass solution to be used sparingly and only in case of emergency.

With this in mind, even if Powertools allowed to change the log level at runtime, the behavior you're observing would still persist. Furthermore, since the new values are applied to a new environment, the current way of setting the logLevel for Logger would suffice.

I'm going to leave the feature request open to give a chance to discover other use cases and hear other customers as well.

From this discussion it appears that there might be other use cases in which changing the log level at runtime might be useful.

[Aaronius]

Thank you for this information @dreamorosi. That seems to match the behavior I'm observing. It does make switching log levels at runtime problematic as you've noted. Thanks for taking my feedback!

[codyfrisch]

I'm going to leave the feature request open to give a chance to discover other use cases and hear other customers as well.

From this discussion it appears that there might be other use cases in which changing the log level at runtime might be useful.

I have another use case for this: dynamically setting the logging level based on a value in the event or context in a multi-tenant environment. Lambda authorizer may set a "debug" level that appears in the event.requestContext.authorizer.lambda.logLevel (for example). I can then "set" the log level for the execution based on that. This way if a specific user is having an issue I can increase logging to troubleshoot just that user without capturing data from all users (who havent authorized us to capture data in logs for example)

[github-actions]

⚠️ COMMENT VISIBILITY WARNING ⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[dreamorosi]

I have just merged a PR that introduces new methods to access and set the log level at runtime (see linked PR for usage examples).

The new feature will be available in the next release!

[github-actions]

This is now released under v1.9.0 version!