Feature request: Clear log buffer before starting next one

[zirkelc]

Hi,

I went through the code of the new log buffering feature and noticed a potential memory issue. The way it is currently implemented, both clearBuffer() and flushBuffer() will reset the log items only for the current X-Ray Trace ID:

clearBuffer

powertools-lambda-typescript/packages/logger/src/Logger.ts

Lines 1386 to 1392 in cab716d

	public clearBuffer(): void {
	const traceId = this.envVarsService.getXrayTraceId();
	if (traceId === undefined) {
	return;
	}
	this.#buffer?.delete(traceId);
	}

flushBuffer

powertools-lambda-typescript/packages/logger/src/Logger.ts

Line 1379 in cab716d

this.#buffer?.delete(traceId);

However, that means if the Logger is created outside of the handler and is not explicitly cleared, or flushed through an error, the buffer will keep logs from previous invocations with different X-Ray Trace IDs. For example, a Lambda instance that runs every few minutes will be re-used and eventually fill up the buffer with log items.

Since the buffer is implemented as a Map, I think calling this.#buffer?.clear() instead of this.#buffer?.delete(traceId) should have the same effect without the potential memory issue.

[dreamorosi]

Hi @zirkelc, this design was intentional however now that you mention it I think we should've called this out in the documentation more clearly.

The logger.clearBuffer() method is called automatically after each invocation/request whenever you use the injectLambdaContext() Middy.js middleware or class method decorator. Because of this, there's no danger of the buffer growing in size.

The only case where this can happen is if you're not using either of the two above, and instead you're using the logger "manually". In that case you're responsible for either flushing it or clearing it manually, i.e.

import { Logger } from '@aws-lambda-powertools/logger';

const logger = new Logger({
  logBufferOptions: { enabled: true }
});

export const logger = async () => {
  logger.debug('a buffered log');

  try {

  } finally {
    logger.clearBuffer();
  }
}

We decided to go in this direction because we don't know yet how our customers will use this feature and while we officially support AWS Lambda as only compute environment, if customers use the logger elsewhere and especially in environments that support concurrent requests, clearing the entire Map would result in data loss.

As I mentioned, I believe we don't call out the fact that you need to clear the buffer when not using any of the two injectLambdaContext(), I'll open a PR to add it to the docs.

[zirkelc]

Hi @dreamorosi

thanks for the quick response. I assumed this behavior could be intentional and the reason for leaving it open definitely makes sense.

However, I think there's a good chance some customers will forget to clear the buffer manually, especially when you break-apart the Lambda handler code into sub-functions and simply re-use the Logger instance from somewhere else.

The Middy.js injectLambdaContext() is a good way to handle it, I personally don't use it for the reason that it adds the context to every log item. I instead log the event and context once at the beginning of the handler. But that's just my personal preference.

Maybe an additional flag like resetAfterFlush: boolean or resetCompleteBuffer: boolean could be added to the logBufferOptions. Setting this flag will clear the entire buffer instead of only clearing the current X-Ray ID. Having such a flag would also help to emphasize the current behavior.

[dreamorosi]

Hi @zirkelc, sorry for the delay - I wanted to wait to discuss this with the team before making a decision.

I think you're right, and we should clear the buffer before the next request.

For now we have decided to make this the default behavior, so in practice you'll be able to have this behavior with no changes besides upgrading to the right version once it's released.

I expect to work on this within the next days, so it can be included in the next release.

[leandrodamascena]

I'll open a similar issue in Python! Thanks for bringing this up @zirkelc

[zirkelc]

Great, thank you!

[github-actions]

⚠️ COMMENT VISIBILITY WARNING ⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.

[github-actions]

This is now released under v2.17.0 version!