Maintenance: evaluate migration to Middy 3.x

[dreamorosi]

Description of the feature request

Problem statement
Middy, the dependency that we use to vend middleware has recently released a new major version (v 3.x). This version drops support for Node JS version 12.x and introduces some breaking changes.

Given that Middy, as a whole, has a large surface and we are only using @middy/core we need to investigate on whether there's a path to upgrade to the newer version while still continuing to support Node JS 12.x in Powertools.

Additional context
Running the e2e tests on a branch that upgraded @middy/core to 3.0.1 seems to show that the newer release is compatible with Powertools on both versions 12 and 14 of Node (see run results).

At the same time, the unit tests for that same branch are failing for all utilities and specifically in the section that relates the middleware implementations. The error message (see detail below) seems to hint at incompatibilities between the new bundling of middy that now supports both CJS and ESM bundling, and our project's configuration since the errors are related to the imports and the tests don't run at all.

 FAIL   AWS Lambda Powertools utility: TRACER  tests/unit/middy.test.ts
  ● Test suite failed to run

    Jest encountered an unexpected token

    Jest failed to parse a file. This happens e.g. when your code or its dependencies use non-standard JavaScript syntax, or when Jest is not configured to support such syntax.

    Out of the box Jest supports Babel, which will be used to transform your files into valid JS based on your Babel configuration.

    By default "node_modules" folder is ignored by transformers.

    Here's what you can do:
     • If you are trying to use ECMAScript Modules, see https://jestjs.io/docs/ecmascript-modules for how to enable it.
     • If you are trying to use TypeScript, see https://jestjs.io/docs/getting-started#using-typescript
     • To have some of your "node_modules" files transformed, you can specify a custom "transformIgnorePatterns" in your config.
     • If you need a custom transformation specify a "transform" option in your config.
     • If you simply want to mock your non-JS modules (e.g. binary assets) you can stub them out with the "moduleNameMapper" config option.

    You'll find more details and examples of these config options in the docs:
    https://jestjs.io/docs/configuration
    For information about custom transformations, see:
    https://jestjs.io/docs/code-transformation

    Details:

    /home/ec2-user/aws-lambda-powertools-typescript/node_modules/@middy/core/index.js:1
    ({"Object.<anonymous>":function(module,exports,require,__dirname,__filename,jest){import{EventEmitter}from"events";const defaultLambdaHandler=()=>{};const defaultPlugin={timeoutEarlyInMillis:5,timeoutEarlyResponse:()=>{throw new Error("Timeout")}};const middy=(lambdaHandler=defaultLambdaHandler,plugin={})=>{if(typeof lambdaHandler!=="function"){plugin=lambdaHandler;lambdaHandler=defaultLambdaHandler}plugin={...defaultPlugin,...plugin};plugin.timeoutEarly=plugin.timeoutEarlyInMillis>0;plugin.beforePrefetch?.();const beforeMiddlewares=[];const afterMiddlewares=[];const onErrorMiddlewares=[];const middy1=(event={},context={})=>{plugin.requestStart?.();const request={event,context,response:undefined,error:undefined,internal:plugin.internal??{}};return runRequest(request,[...beforeMiddlewares],lambdaHandler,[...afterMiddlewares],[...onErrorMiddlewares],plugin)};middy1.use=middlewares=>{if(!Array.isArray(middlewares)){middlewares=[middlewares]}for(const middleware of middlewares){const{before,after,onError}=middleware;if(!before&&!after&&!onError){throw new Error('Middleware must be an object containing at least one key among "before", "after", "onError"')}if(before)middy1.before(before);if(after)middy1.after(after);if(onError)middy1.onError(onError)}return middy1};middy1.before=beforeMiddleware=>{beforeMiddlewares.push(beforeMiddleware);return middy1};middy1.after=afterMiddleware=>{afterMiddlewares.unshift(afterMiddleware);return middy1};middy1.onError=onErrorMiddleware=>{onErrorMiddlewares.unshift(onErrorMiddleware);return middy1};middy1.handler=replaceLambdaHandler=>{lambdaHandler=replaceLambdaHandler;return middy1};return middy1};const runRequest=async(request,beforeMiddlewares,lambdaHandler,afterMiddlewares,onErrorMiddlewares,plugin)=>{const timeoutEarly=plugin.timeoutEarly&&request.context.getRemainingTimeInMillis;try{await runMiddlewares(request,beforeMiddlewares,plugin);if(request.response===undefined){plugin.beforeHandler?.();const handlerAbort=new AbortController;let timeoutAbort;if(timeoutEarly)timeoutAbort=new AbortController;request.response=await Promise.race([lambdaHandler(request.event,request.context,{signal:handlerAbort.signal}),timeoutEarly?setTimeoutPromise(request.context.getRemainingTimeInMillis()-plugin.timeoutEarlyInMillis,{signal:timeoutAbort.signal}).then(()=>{handlerAbort.abort();return plugin.timeoutEarlyResponse()}):Promise.race([])]);if(timeoutEarly)timeoutAbort.abort();plugin.afterHandler?.();await runMiddlewares(request,afterMiddlewares,plugin)}}catch(e){request.response=undefined;request.error=e;try{await runMiddlewares(request,onErrorMiddlewares,plugin)}catch(e){e.originalError=request.error;request.error=e;throw request.error}if(request.response===undefined)throw request.error}finally{await plugin.requestEnd?.(request)}return request.response};const runMiddlewares=async(request,middlewares,plugin)=>{for(const nextMiddleware of middlewares){plugin.beforeMiddleware?.(nextMiddleware.name);const res=await nextMiddleware(request);plugin.afterMiddleware?.(nextMiddleware.name);if(res!==undefined){request.response=res;return}}};const polyfillAbortController=()=>{if(process.version<"v15.0.0"){let _toStringTag;let _toStringTag1=Symbol.toStringTag;class AbortSignal{toString(){return"[object AbortSignal]"}get[_toStringTag1](){return"AbortSignal"}removeEventListener(name,handler){this.eventEmitter.removeListener(name,handler)}addEventListener(name,handler){this.eventEmitter.on(name,handler)}dispatchEvent(type){const event={type,target:this};const handlerName=`on${type}`;if(typeof this[handlerName]==="function")this[handlerName](event);this.eventEmitter.emit(type,event)}constructor(){this.eventEmitter=new EventEmitter;this.onabort=null;this.aborted=false}}return _toStringTag=Symbol.toStringTag,class AbortController{abort(){if(this.signal.aborted)return;this.signal.aborted=true;this.signal.dispatchEvent("abort")}toString(){return"[object AbortController]"}get[_toStringTag](){return"AbortController"}constructor(){this.signal=new AbortSignal}}}else{return AbortController}};global.AbortController=polyfillAbortController();const polyfillSetTimeoutPromise=()=>{return(ms,{signal})=>{if(signal.aborted){return Promise.reject(new Error("Aborted","AbortError"))}return new Promise((resolve,reject)=>{const abortHandler=()=>{clearTimeout(timeout);reject(new Error("Aborted","AbortError"))};const timeout=setTimeout(()=>{resolve();signal.removeEventListener("abort",abortHandler)},ms);signal.addEventListener("abort",abortHandler)})}};const setTimeoutPromise=polyfillSetTimeoutPromise();export default middy
                                                                                      ^^^^^^

    SyntaxError: Cannot use import statement outside a module

       6 |
       7 | import { captureLambdaHandler } from '../../src/middleware/middy';
    >  8 | import middy from '@middy/core';
         | ^
       9 | import { Tracer } from './../../src';
      10 | import type { Context, Handler } from 'aws-lambda/handler';
      11 | import { Segment, setContextMissingStrategy, Subsegment } from 'aws-xray-sdk-core';

      at Runtime.createScriptFromCode (../../node_modules/jest-runtime/build/index.js:1728:14)
      at Object.<anonymous> (tests/unit/middy.test.ts:8:1)

Code examples
N/A

Benefits for you and the wider AWS community

Describe alternatives you've considered
Not upgrading to 3.x and staying on version 2.5.x

Additional context

Related issues, RFCs

#861

[willfarrell]

Middy does not support nodejs12.x out of the box, but @middy/core​ with an additional build step to transpile newer syntax down to es2019 would support running on nodejs12.x. I just tested this using es2019​ in our build config with nodejs12.x locally and it tested without issue for @middy/core.

The breaking changes around onError does not affect any middleware within powertools from what I can see.

v2.5.x is very stable and is fine to stay with till AWS itself deprecates nodejs12.x. Because we're community maintained, I have no plans to apply any future updates to that branch. However, I would make an exception for security patches. I don't expect any though because @middy/core​ doesn't have any runtime dependencies by design and we're passing CodeQL with flying colours currently.

[shooit]

Anecdotally, we are using @middy/core at v3.0.3 with the Powertools middleware with no issues. However we run all our lambdas on Node 16.x so we are not affected by Middy's deprecation of Node 12.x.

[dreamorosi]

I can confirm that at least on my experience @middy/core v3.x doesn't cause issues at runtime.

However I have tried upgrading the dependency in the repo and I have found that it makes the unit tests ran on Node.js 12 fail. At the time of writing we run unit and integration tests on Node.js 12, 14, and 16.

We'll revisit the issue we decide to drop Node.js 12, to track the progress of this decision follow #1061

[dreamorosi]

While migrating the codebase to Node.js 18.x, we have updated the devDependencies and as a part of that effort we have updated @middy/core to version 3.x, which si now also part of our CI tests.

We can consider this issue as complete, we will revisit the Middy version once we drop support for Node.js 14.

[github-actions]

⚠️ COMMENT VISIBILITY WARNING ⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.