Feature Request - add support to other observability providers

[caioquirino]

Note

See this comment for latest status update. If you're interested in seeing this come to fruition, please consider adding a 👍 to this issue.

---

Description of the feature request

Problem statement
We have used the js version of lambda-powertools for a long time already in DAZN, and now we have most of the Back-end teams working with typescript instead of javascript directly. With that said, the interest in adopting this project is really high, and it is useful for many areas already. In order to make the best adoption of this amazing tool in DAZN, it would be great if we could either configure it to use New Relic instead of Cloudwatch, or extend it with our own implementation, so we could keep it self-contained in a single repository.

Currently we are using New Relic for both logging, metrics and tracing in practically all the services in DAZN.

Summary of the feature
Provide a way to configure or extend this tools for external providers (New Relic)

Example suggestions:

Logger options:

• Add correspondent middlewares for New Relic
• Propose a way to configure the middlewares to use a New Relic implementation
• Add some extensibility in the Logger so it's possible to override the printLog function (or something like that) or provide an alternative implementation to the Logger class

Metrics options:

• Add correspondent middlewares for New Relic
• Propose a way to configure the middlewares to use a New Relic Implementation
• Add the ability to provide an alternative implementation to the Metrics class

Tracing options:

• Add correspondent middlewares for New Relic
• Propose a way to configure the middlewares to use a New Relic Implementation
• Add the ability to provide an alternative implementation to the Tracer class

Code examples
To be defined

Benefits for you and the wider AWS community
Developers that needs to use New Relic or any other observability provider are able to use this project

Describe alternatives you've considered
It's briefly documented in the Summary, still needs more understanding of which options are possible

Additional context

Related issues, RFCs

aws-powertools/powertools-lambda-python#1433

[dreamorosi]

Hi @caioquirino thank you for opening this issue and formalizing the request!

As we have mentioned during our meeting with the Backend Guild this is something that we are keen to consider.

Please allow us some time to discuss this internally and we'll be back to you and the team.

[saragerion]

Thanks for opening this issue @caioquirino!

[dreamorosi]

Hi everyone, we have started working on a RFC for this topic and the first tests are showing that we might be able to support 3rd party observability providers by allowing you to fully customize the log formatter used to emit logs.

If you're interested in reading the current proposal and leaving your feedback please do so at #1500.

[dreamorosi]

As we get into 2025, I'd like to share an update on this topic.

Current status

As of today out of our three core utilities, Logger is the only one that can work with different providers. This module is not concerned about where you send your logs but only about how you emit them. For this reason we invested time into making it as configurable as possible and as far as we know, customers are using it both with Amazon CloudWatch and other providers. Generally speaking we expect customers using Logger to send logs to 3rd party providers using AWS Lambda Extensions rather a specific provider's SDK.

For the other two, Metrics emits metrics only in Amazon Embedded Metrics Format (EMF) but, like Logger, is not concerned with how you send these blobs and just emits them to stdout. Powertools for AWS Lambda (Python) added a DataDog-specific implementation but for now we have no plans of following suit in this version of Powertools. With this in mind, unless your provider supports EMF and you're able to send these metrics using the Telemetry API exposed by Lambda, we acknowledge that the utility of this feature might be limited to you.

Finally, when it comes to Tracer, of the three it's the one that is more coupled with AWS as it directly depends on the AWS X-Ray SDK and such is compatible only with AWS X-Ray.

What's on our radar

Our current line of thinking is that we'd like to avoid adding vendor-specific implementations and take a dependency on these SDKs unless there's a very compelling reason (aka strong customer demand).

With this in mind, we're instead investigating opportunities to support Open Telemetry in all three core utilities. We believe that adopting this standard will help us get closer to the goal of helping customers adopt Powertools for AWS in any workload, even if it doesn't involve sending telemetry data to CloudWatch.

This is going to be a long term project that we will pursue during 2025 and that we hope to share more info about as it's ready.

---

Finally, as closing comment, I just wanted to acknowledge that the issue is likely too broad to be actionable, however I will leave it open & rename it as a mean to gather interest on adding support to 3rd party observability providers rather than specifically New Relic.