Feature request: extend API Gateway schemas with custom schema in `body` rather than `string`

[julbrs]

Expected Behavior

I expect the parsing event document to work: https://docs.powertools.aws.dev/lambda/typescript/latest/utilities/parser/#parse-events

Current Behavior

When I implement such code, I ended with the error:

ParseError: Failed to parse schema.

Because no envelope was set.

Code snippet

My failing code:

import { newNetsuiteUserSchema } from '@/packages/core/model/netsuite';

import middy from '@middy/core';
import { parser } from '@aws-lambda-powertools/parser/middleware';
import { APIGatewayProxyEventV2Schema } from '@aws-lambda-powertools/parser/schemas';
import { z } from 'zod';

export const baseHandler = async (event: z.infer<typeof newNetsuiteUserSchema>) => {
  return {
    statusCode: 200,
    body: JSON.stringify(event),
  };
};

export const handler = middy(baseHandler).use(parser({ schema: newNetsuiteUserSchema }));

with:

export const newNetsuiteUserSchema = z.object({
  firstName: z.string().min(1),
  lastName: z.string().min(1),
  email: z.string().email().min(1),
  role: z.nativeEnum(UserRoles),
  custentity_lba_customer_center_access: z.string().min(1),
});

Steps to Reproduce

I am using SST (sst.dev) to build my project.

Possible Solution

I think it's mandatory to specify the envelope like that:

import { newNetsuiteUserSchema } from '@/packages/core/model/netsuite';

import middy from '@middy/core';
import { parser } from '@aws-lambda-powertools/parser/middleware';
import { APIGatewayProxyEventV2Schema } from '@aws-lambda-powertools/parser/schemas';
import { z } from 'zod';
import { ApiGatewayV2Envelope } from '@aws-lambda-powertools/parser/envelopes';

export const baseHandler = async (event: z.infer<typeof newNetsuiteUserSchema>) => {
  return {
    statusCode: 200,
    body: JSON.stringify(event),
  };
};

export const handler = middy(baseHandler).use(
  parser({ schema: newNetsuiteUserSchema, envelope: ApiGatewayV2Envelope })
);

Powertools for AWS Lambda (TypeScript) version

latest (2.5.0-beta)

AWS Lambda function runtime

20.x

Packaging format used

npm

Execution logs

},
     {
       "expected": "'Admin' | 'User'",
       "received": "undefined",
       "code": "invalid_type",
       "path": [
         "role"
       ],
       "message": "Required"
     },
     {
       "code": "invalid_type",
       "expected": "string",
       "received": "undefined",
       "path": [
         "custentity_lba_customer_center_access"
       ],
       "message": "Required"
     }
   ].
       at parse (/Users/julien/work/lab/customer-center/node_modules/.pnpm/@aws-lambda-powertools+parser@2.5.0-beta_@middy+core@5.4.3_zod@3.23.8/node_modules/@aws-lambda-powertools/parser/lib/esm/parser.js:41:15)
       at before (/Users/julien/work/lab/customer-center/node_modules/.pnpm/@aws-lambda-powertools+parser@2.5.0-beta_@middy+core@5.4.3_zod@3.23.8/node_modules/@aws-lambda-powertools/parser/lib/esm/middleware/parser.js:33:25)
       at runMiddlewares (/Users/julien/work/lab/customer-center/node_modules/.pnpm/@middy+core@5.4.3/node_modules/@middy/core/index.js:230:23)
       at runRequest (/Users/julien/work/lab/customer-center/node_modules/.pnpm/@middy+core@5.4.3/node_modules/@middy/core/index.js:154:11)
       at middyHandler (/Users/julien/work/lab/customer-center/node_modules/.pnpm/@middy+core@5.4.3/node_modules/@middy/core/index.js:43:12)
       at file:///Users/julien/work/lab/customer-center/node_modules/.pnpm/sst@2.43.4_@aws-sdk+client-sso-oidc@3.606.0_@aws-sdk+client-sts@3.606.0_aws-crt@1.21.3__aws-c_gdj3g64c23iicrgcjjtaychpzi/node_modules/sst/support/nodejs-runtime/index.mjs:148:22
       at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #typescript channel on our Powertools for AWS Lambda Discord: Invite link

[dreamorosi]

Hi @julbrs, thank you for opening this issue.

Based on the description and logs, it seems that the utility is working as expected.

When you specify a schema, the event will be validated using that schema as-is. In your case, based on your schema, the event that would allow your handler to pass validation would be something like:

{
  "firstName": "foo",
  "lastName": "bar",
  "email": "foo.bar@example.com",
  "role": "guest",
  "custentity_lba_customer_center_access": "baq",
}

In your example however, you included the APIGatewayProxyEventV2Schema, which suggests you're using Amazon API Gateway with AWS Lambda proxy integration.

If this is the case, the event will look something like this:

{
  "version": "2.0",
  "routeKey": "POST /lambda",
  "rawPath": "/lambda",
  "rawQueryString": "",
  "headers": {
    "accept": "*/*",
    "accept-encoding": "gzip, deflate",
    "authorization": "Bearer foo",
    "content-length": "0",
    "host": "lsw1ro4ipb.execute-api.eu-west-1.amazonaws.com",
    "user-agent": "HTTPie/3.2.2",
    "x-amzn-trace-id": "Root=1-66705bc7-2b4257df30cbee696ef2cf28",
    "x-forwarded-for": "15.248.3.126",
    "x-forwarded-port": "443",
    "x-forwarded-proto": "https"
  },
  "requestContext": {
    "accountId": "123456789012",
    "apiId": "lsw1ro4ipb",
    "authorizer": {
      "lambda": null
    },
    "domainName": "lsw1ro4ipb.execute-api.eu-west-1.amazonaws.com",
    "domainPrefix": "lsw1ro4ipb",
    "http": {
      "method": "POST",
      "path": "/lambda",
      "protocol": "HTTP/1.1",
      "sourceIp": "15.248.3.126",
      "userAgent": "HTTPie/3.2.2"
    },
    "requestId": "ZhNHJhhLjoEEPiw=",
    "routeKey": "POST /lambda",
    "stage": "$default",
    "time": "17/Jun/2024:15:52:39 +0000",
    "timeEpoch": 1718639559080
  },
  "isBase64Encoded": false,
  "body": "....." <-- your payload goes here
}

With this in mind, you have two options:

1. Use the envelope - like you did
2. Extend the APIGatewayProxyEventV2Schema schema

const mySchema = APIGatewayProxyEventV2Schema.extend({
  body: newNetsuiteUserSchema
});

In both cases Parser will validate both the outer object (aka the envelope - in this case the API Gateway HTTP Proxy event) and the inner payload (aka the body with the schema you provide). The main difference between the two is that the envelope one replaces the event and gives you only your data (in this case the parsed body) while the other gives you the entire event including also all the other fields (i.e. headers, queryString, etc.).

[julbrs]

Hi @dreamorosi,

Thanks for the time to understand my issue. I confirm that I am using a Lambda behind API Gateway v2 so my event object contains much more content that only my payload, and the payload is in fact only in the body field.

Regarding schema extension, I think it can be a nice solution. I have also tried that but I cannot reproduce the "magic" generated by the Envelope (that is converting the string body to a parsed object!).

Here is an example:

import { newNetsuiteUserSchema } from '@/packages/core/model/netsuite';

import middy from '@middy/core';
import { parser } from '@aws-lambda-powertools/parser/middleware';
import { APIGatewayProxyEventV2Schema } from '@aws-lambda-powertools/parser/schemas';
import { z } from 'zod';

// extended APIGatewayProxyEventV2Schema
export const newNetsuiteUserEventSchema = APIGatewayProxyEventV2Schema.extend({
  body: newNetsuiteUserSchema,
});

export const baseHandler = async (event: z.infer<typeof newNetsuiteUserEventSchema>) => {
  console.log(event);
  return {
    statusCode: 200,
    body: JSON.stringify(event),
  };
};

export const handler = middy(baseHandler).use(parser({ schema: newNetsuiteUserEventSchema }));

Here when I am calling my API, I end with:

Error: Failed to parse schema. This error was caused by: [
  {
    "code": "invalid_type",
    "expected": "object",
    "received": "string",
    "path": [
      "body"
    ],
    "message": "Expected object, received string"
  }
].

This is expected because the body is a stringify version of my payload, not an object ;)

I think I need to provide a custom Envelope here to the parser fonction, so the body will be correctly parsed, am I right?

I found the API Gateway v2 Envelope here: https://github.com/aws-powertools/powertools-lambda-typescript/blob/e709c9c24b9e45a5b8850c35f760cdec2fb43c6e/packages/parser/src/envelopes/apigwv2.ts

[dreamorosi]

I see what you mean now, you're right we don't do any transform when extending the body of a schema.

I was wrong in my previous statement, I mixed the schemas and the envelopes and assumed we were doing the same in both. In my mind, I had the example of EventBridgeSchema which we can extend because the details is an object itself, but in this case the body is a stringified representation of the JSON object.

In this case, the way you'd extend the schema would look something like this:

export const newNetsuiteUserEventSchema = APIGatewayProxyEventV2Schema.extend({
  body: z
    .string()
    .transform((str, ctx) => {
      try {
        return JSON.parse(str);
      } catch (e) {
        ctx.addIssue({ code: 'custom', message: 'Invalid JSON' });
        return z.NEVER;
      }
    })
    .pipe(newNetsuiteUserSchema),
});

I tested the above with the copy/pasted version of your schema and function and the parsing is successful.

I guess the next question would be whether we should add this by default or not. I can see how without it, the schema would require a second parsing within the function handler just for the body - assuming one doesn't want to use the envelope, for example because they want to retain access to fields other than the body.

I'm going to convert this to a feature request, and regardless of the decision we make on adding it, we should likely consider updating the documentation to include this example.

My main concern here is, how to integrate this in a way that customers can still use the plain string if they don't want to specify a schema AND also how to make the .extend() cleaner (i.e. without asking people to use .pipe(mySchema).

Would like to loop @am29d here, since he's the Zod expert.