Feature request: export Records schema building blog for customizations

[am29d]

Use case

When you want to extend the built-in schema with your custom schema you can extend the it and overwrite specific parts with of the built-in schema we provide.

const mySchema = z.object({
  name: z.string(),
  age: z.number()
});

const AlbCustomizedSchema = AlbSchema.extend({
  body: mySchema
});

The body value is z.string() in the AlbSchema and we can plug in modifications.

This works for schemas with first the modifiable attribute available in the main schema, such as AlbSchema, EventBrdigeSchema or others.

When working with schemas that contain a list of elements the body part is not accessible, thus customers can't extend it without replicating the entire Record. For example, the SqsEvent is:

  {
    "Records": [
      {
        "messageId": "059f36b4-87a3-44ab-83d2-661975830a7d",
        "receiptHandle": "AQEBwJnKyrHigUMZj6rYigCgxlaS3SLy0a...",
        "body": "Test message.",
        "attributes": {
          "ApproximateReceiveCount": "1",
          "SentTimestamp": "1545082649183",
          "SenderId": "AIDAIENQZJOLO23YVJ4VO",
          "ApproximateFirstReceiveTimestamp": "1545082649185"
        },
        "messageAttributes": {
          "testAttr": {
            "stringValue": "100",
            "binaryValue": "base64Str",
            "dataType": "Number"
          }
        },
        "md5OfBody": "e4e68fb7bd0e697a0ae8f1bb342846b3",
        "eventSource": "aws:sqs",
        "eventSourceARN": "arn:aws:sqs:us-east-2:123456789012:my-queue",
        "awsRegion": "us-east-2"
      }
    ]
  }

To overwrite the body, we'd need:

const SqsCustomizedSchema = SqsSchema.extend({
 Records: z.array(
     z.object({
         messageId...
         ...
         body: mySchema,
         // the rest
     })
  )
})

Solution/User Experience

Export the record elements that are nested parts of the built-in schema. For SQS it would be

powertools-lambda-typescript/packages/parser/src/schemas/sqs.ts

Lines 26 to 38 in 1e7175a

	const SqsRecordSchema = z.object({
	messageId: z.string(),
	receiptHandle: z.string(),
	body: z.string(),
	attributes: SqsAttributesSchema,
	messageAttributes: z.record(z.string(), SqsMsgAttributeSchema),
	md5OfBody: z.string(),
	md5OfMessageAttributes: z.string().optional().nullable(),
	eventSource: z.literal('aws:sqs'),
	eventSourceARN: z.string(),
	awsRegion: z.string(),
	});

This would allow us change only the body field

const SqsMySchema = SqsSchema.extend({
  Records: SqsRecordSchema.extend({
   body: mySchema
  })
})

The requirement for nested Records exports applies to:

• KafkaRecordSchema
• KinesisFirehoseRecrod
• KinesisFirehoseSqsSchema
• KinesisDataStreamRecord
• SesRecordSchema
• SnsRecordSchema and SnsNotificationSchema
• SqsRecordSchema

While some of this objects are exported in their own files, they are not part of the barrel export and thus customers can't import them.

Alternative solutions

No response

Acknowledgment

• This feature request meets Powertools for AWS Lambda (TypeScript) Tenets
• Should this be considered in other Powertools for AWS Lambda languages? i.e. Python, Java, and .NET

Future readers

Please react with 👍 and your use case to help us understand customer demand.

[dreamorosi]

Hi @am29d, this is a good point and we should definitely expose these schemas.

My only concern is with where to export them. On one hand I see the value of exposing them in the main barrel file both for consistency and relative ease of usage. On the other hand, I think we should help customers do the right thing and instead provide scoped exports.

Even when using ESM with tree-shaking enabled, using the barrel file causes all the schemas to be pulled in into the bundle. This is because esbuild cannot know which one will actually be used at runtime, it only "sees" that you've required the entire barrel file. The same is true for envelopes.

Exporting schemas in their own subpath export like we started doing with API Gateway-related schemas would instead allow customers to target a specific group of schemas and bring in only these.

I don't have concrete data to back this up, but I would imagine that customers won't be using more than a handful built-in schemas in each function. If this is true, then giving scoped exports will help to keep their bundle size small, especially as we add more and more specialized schemas.

So in short, I am in favor of exporting these schemas as they improve the DX, but I would suggest we do this responsibly and provide subpath exports for each service.

[am29d]

I agree, I have seen your changes for API gateway and we should change other exports as well, i.e.:

before

import { SqsSchema } from '@aws-lambda-powertools/parser/schemas';

after

import { SqsSchema } from '@aws-lambda-powertools/parser/schemas/sqs';

[dreamorosi]

As suggested in #2829 by @am29d, this issue & that one are very much related.

I think it'd be useful to complete this before merging that one, so we can include the new schemas in the subpath exports we'll add in #2829.

[daschaa]

@am29d @dreamorosi Should i take this issue first then? :)

[am29d]

@am29d @dreamorosi Should i take this issue first then? :)

@daschaa of course, all yours :)

[github-actions]

⚠️ COMMENT VISIBILITY WARNING ⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.

[github-actions]

This is now released under v2.7.0 version!