Maintenance: add mechanism to check compatibility with @types/aws-lambda package

[am29d]

Summary

When using our built-in schema with zod, developers can infer the types with type SqsEvent = z.infer<typeof SqsSchema> and use it in their Lambda handler. Another popular option is to use @types/aws-lambda package wich has 3M downloads per week. We need to make sure that both types are compatible, so there is no confusion or unexpected behaviour.

Why is this needed?

If either @types/aws-lambda or our schema is updated, we want to have a reliable test to verify that they are both compatible and if not to fix one them.

Which area does this relate to?

Parser

Solution

I have checked the first idea that worked:

import { z } from 'zod';
import { ALBEvent, APIGatewayProxyEvent } from 'aws-lambda';
import { AlbSchema } from '../../src/schemas/alb';
import { APIGatewayProxyEventSchema } from '../../src/schemas/apigw';

type AlbType = z.infer<typeof AlbSchema>;
type ApiGatewayType = z.infer<typeof APIGatewayProxyEventSchema>;
describe('Types compatibility', () => {
  it('ALBEvent types are compatible', () => {
    // These functions will fail to compile if A and B are not structurally identical
    // eslint-disable-next-line @typescript-eslint/no-unused-vars
    const A = (value: AlbType): ALBEvent => value;
    // eslint-disable-next-line @typescript-eslint/no-unused-vars
    const B = (value: ALBEvent): AlbType => value;
  });

  it('ApiGatewayEvent types are compatible', () => {
    // eslint-disable-next-line @typescript-eslint/no-unused-vars
    const A = (value: ApiGatewayType): APIGatewayProxyEvent => value;
    // eslint-disable-next-line @typescript-eslint/no-unused-vars
    const B = (value: APIGatewayProxyEvent): ApiGatewayType => value;
  });
});

The compiler will fail if the structure does not match and will tell exactly which properties are not compatible.

Alternatively research more about https://github.com/fabien0102/ts-to-zod. If we can convert @types/aws-lambda to schemas, we might directly compare the schemas to find the differences.

Acknowledgment

• This request meets Powertools for AWS Lambda (TypeScript) Tenets
• Should this be considered in other Powertools for AWS Lambda languages? i.e. Python, Java, and .NET

Future readers

Please react with 👍 and your use case to help us understand customer demand.

[dreamorosi]

Copying this comment I have left under a PR since it's relevant to the conversation:

---

[...] I think we should agree & articulate our stance on this topic a bit better to avoid future confusion amongst us and towards customers.

The main point here, besides calling out potential discrepacies between us & the other library, is in my opinion to establish our strategy towards event types as a whole. This will then influence which actions we'll take now and in the future, as well as when discrepacies are detected.

Currently the @types/aws-lambda types are community maintained and manually authored. AWS events can change and the types can contain issues.

While I agree with the sentiment of maintaining these types and our schemas (and in the future JSON schemas as well) up to date and aligned, I don't think this ultimately guarantees that the types/schemas are actually correct.

As a think big idea, we should at least consider the option of instead investing efforts in creating automations to periodically test against real AWS events to get the actual shapes of the events sent to Lambda. Based of those, we could then derive the schemas as well as the TypeScript types.

Once we detect drift (aka changes in the events), we could then propagate them to the appropriate destinations be it within the scope of Powertools or @types/aws-lambda.

I am not expecting the whole industry to move over our own types (i.e. @aws-lambda-powertools/events or @aws-lambda-powertools/types) but I would still aim at becoming the source of truth for these events/schemas/types.

The DefinitelyTyped repo (aka the one that hosts @types/* packages) already has precedents of depending on external type definition, one of them was added in the past 6 months to the @types/node types to support the fetch module. This module was originally implemented under undici and later incorporated into Node.js. The source still resides in the undici repo, so to add support for these types they started publishing type definitions for under undici-types that are now a direct depenencey pf the @types/node package (see here).

While it'll be a lot of work, I believe this would bring a lot of value to customers and eventually would help with establishing a single and authoritative source of truth for AWS events that today does not exist. Doing this would benefit not only Powertools TS or the Node.js community but also Powertools Python and the Python community as a whole via Pydantic. Additionally, it would open the door to future use cases like testing factories (#445), integrations with SAM or CDK, and similar.

[dreamorosi]

Over the past ~year adoption of the Parser utility has been good and thanks to customer feedback and issue reports we have been able to improve our schemas so that they are serviceable.

We are likely a long way from being able to test live resources and the discussion is larger than what can be summarized in an issue, as well as involving topics that we need to resolve internally before being able to do so.

For the time being I will close this issue as not planned, however I encourage anyone reading this issue to please continue reporting issues if you see any schema that is not accurate.

Additionally, if anyone feels strongly about us stepping up and providing a comprehensive event portfolio tested on live events, please do send us an email from your work email at [email protected] - the option is still on the table but we need significant traction from our customers.

[github-actions]

⚠️ COMMENT VISIBILITY WARNING ⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.