feat(gatsby): Adapters (#38232)

* initial - cache restoration

* snapshot

* cache.restore() return comment

* redirect status

* note about named wildcard paths

* don't pass named wildcards to function manifest

* fix status code resolution for redirects

* scaffold initial gatsby-adapter-netlify package and use if for dev purposes (hardcoded for now)

* build adapter package as ESM and load it as such, so we can use ESM-only utils - like @netlify/cache-utils

* no require in esm world

* add redirect headers

* start scaffolding autoinstallation of adapters

* webpack assets + unmanaged assets (start)

* static queries, app-data.json, minor refactors and initial setup for having routes sorted by specificity

* move adapter version to 1.0.0

* generalize get-latest-apis for adapters

* handle JS files in get-latest-gatsby-files

* set peerDep

* use other testing pkg

* get installation, discovery, re-using working

* update versions

* move adapter init into its own file

* add version checking

* adjust comment

* move stuff around

* feat: add headers to gatsby-config

* misc stuff

* initial engine lambda

* start headers utils

* update deps

* rewrite util

* linting

* convert to obj args

* remove todo comment

* add requiredFiles to functions manifest

* make headers default to []

* move constants to own file

* export rankRoute

* delete unneeded util

* createHeadersMatcher initial impl

* use createHeadersMatcher

* fix types

* add http status code type

* improve createHeadersMatcher and add tests

* move adapterManager init to initialize func

* adjust func args to move reporter and allow adapter options

* add "adapter" option to gatsby-config

* put netlify adapter first in the list - it will only match when env var is set, so won't change default behavior of using testing one

* kebabcase function name as function id

* export FunctionDefinition type

* req.path -> req.url

* initial functions wrapping/bundling in gatsby-adapter-netlify

* remove netlify adapter from gatsby deps, add gatsby as devDep to adapter (to access types)

* fix bundling function files containing [ ]

* unify tsconfig for adapter

* add joi testing for adapter setting

* typescript: make bootstrap work again

* generate redirect/rewrite rules, generate 2 variants of  ssr-engine (odb and regular)

* move routes manifest handling into its own module

* generate _headers rules

* add sorting to routesManifest

* adjust graphql-engine bundling to not leave unreasolvable imports

* ssr lambda handling when it executes in read-only dir (use tmpdir() then)

* inject functions matchPath into function bundle and generate req.params inside of it

* serve api from path prefixed path as well

* add path prefix stripping in function wrapper

* add cache store and restore in gatsby-adapter-netlify

* adjust internal 'lambda' name to 'function'

* format lambda-handler

* misc changes

* missing rename

* compile gatsby-adapter-netlify to cjs

* add generator field

* use netlify adpter when NETLIFY or NETLIFY_LOCAL env var is defined

* use headers from config for ssg/dsg

* allow specyfing different lmdb binary than current process, use abi83 if adapters are used (it works on node14, 16 and 18)

* get-route-path tests

* manager refactoring + typo fix

* initialize adapters e2e test

* cypress: remove viewPort configs

* use next preid for netlify adapter

* remove test adapter from adapters manifest

* don't log errors when testing for user installed adapters

* update adapters manifest

* cleanup gatsby-adapter-netlify a bit, add more public adapter related types to gatsby

* e2e: update gitignore

* resolve lmdb binary from lmdb package and not hardcode the forced path

* gatsby-plugin-image add downlevelIteration

* fix persisted redux keys

* update snapshot and mocks

* fix: only run adapters during gatsby build

* resolve netlify functions runtime deps from adapter context

* improve public typings

* adding adapter to e2e test so that dev-cli copies stuff over

* update babel-preset-gatsby-package dep

* e2e: test functions and assets

* e2e: client-only WIP

* e2e: improve basics

* e2e: improve client-only

* e2e: redirects

* merge _headers and _redirects instead of overwriting it

* apply trailing slash option + pass through trailingSlash & pathPrefix

* add unit tests for manager

* improve manager tests

* update types

* improve e2e tests

* add excludeDatastoreFromEngineFunction flow

* normalize path after globbing

* mock shouldBundleDatastore

* rename adapter config types to be less confusing with gatsby-config

* keep same obfuscated path between builds

* normalize more paths

* support custom 404/500 page for serverless functions

* update snapshot

* generate relative imports in function

* skip trying to copy data to tmp if we are downloading from cdn

* improve TS types

* mock uuid

* put requiredFiles in correct place heh

* typo

* snapshot

* handle GATSBY_EXCLUDE_DATASTORE_FROM_BUNDLE env var

* improve README & update types

* code block

* handle partytown routes

* handle slices (html and slice-data)

* handle chunk-map and webpack.stats

* feat: add name to functionsManifest & displayName to Netlify

* update snapshot

* rename headers constants

* handle image-cdn and file-cdn

* update routesManifest test fixture and snapshot

* don't log unmanaged static assets anymore

* handle some TODOs

* add 'adapters' to feature list

* tmp: make peer dependency allow to use with canaries

* feat: add 'supports' to config to let adapters provide some capabilities and potentially fail builds with clear explanation instead of producing faulty deploy

* adjust some text

* apply trailingSlash to tests and other stuff, add utils

* readme and package.json update

* fix ts versions

* enable typecheck for adapters.js

* don't try to use install adapter if no version matches, install version specified in adapters.js

* allow adapter to disable prior deployment plugins

* disable gatsby-plugin-netlify when using gatsby-adapter-netlify

* fix ts

* handle non-alpha-numerc paths

* verbose log adapter package that is being installed

* only try to restore cache if payload provided

* memoize cache utils

* log adapter version

* handle body parsing in produced api functions

* properly handle cases when body parsing already happened and when there is no function handler export

* handle body parsing config in generated function

* drop unused

* remove redirects created by previous deployment plugins

* properly handle default body for status responses in api functions

* put some dev logs as verbose to limit regular terminal output

* put some dev logs as verbose to limit regular terminal output vol2

* maybe deflake function tests

* maybe precompile api functions in develop function tests

---------

Co-authored-by: Michal Piechowiak <[email protected]>

Author: LekoArts

.circleci/config.yml

@@ -444,6 +444,15 @@ jobs:
       - store_test_results:
           path: e2e-tests/trailing-slash/cypress/results
 
+  e2e_tests_adapters:
+    <<: *e2e-executor
+    steps:
+      - run: echo 'export CYPRESS_RECORD_KEY="${CY_CLOUD_ADAPTERS}"' >> "$BASH_ENV"
+      - e2e-test:
+          test_path: e2e-tests/adapters
+      - store_test_results:
+          path: e2e-tests/adapters/cypress/results
+
   starters_validate:
     executor: node
     steps:
@@ -594,6 +603,8 @@ workflows:
           <<: *e2e-test-workflow
       - e2e_tests_trailing-slash:
           <<: *e2e-test-workflow
+      - e2e_tests_adapters:
+          <<: *e2e-test-workflow
       - e2e_tests_development_runtime_with_react_18:
           <<: *e2e-test-workflow
       - e2e_tests_production_runtime_with_react_18:

e2e-tests/adapters/.gitignore

@@ -0,0 +1,13 @@
+node_modules/
+.cache/
+public
+
+# Local Netlify folder
+.netlify
+
+# Cypress output
+cypress/videos/
+cypress/screenshots/
+
+# Custom .yarnrc file for gatsby-dev on Yarn 3
+.yarnrc.yml

e2e-tests/adapters/README.md

@@ -0,0 +1,27 @@
+# adapters
+
+E2E testing suite for Gatsby's [adapters](http://www.gatsbyjs.com/docs/how-to/previews-deploys-hosting/adapters/) feature.
+If possible, run the tests locally with a CLI. Otherwise deploy the site to the target platform and run Cypress on the deployed URL.
+
+Adapters being tested:
+
+- [gatsby-adapter-netlify](https://github.com/gatsbyjs/gatsby/tree/master/packages/gatsby-adapter-netlify)
+
+## Usage
+
+- To run all tests, use `npm run test`
+- To run individual tests, use `npm run test:%NAME` where `test:%NAME` is the script, e.g. `npm run test:netlify`
+
+If you want to open Cypress locally as a UI, you can run the `:debug` scripts. For example, `npm run test:netlify:debug` to test the Netlify Adapter with Cypress open.
+
+### Adding a new adapter
+
+- Add a new Cypress config inside `cypress/configs`
+- Add a new `test:` script that should run `start-server-and-test`. You can check what e.g. `test:netlify` is doing.
+- Run the Cypress test suites that should work. If you want to exclude a spec, you can use Cypress' [excludeSpecPattern](https://docs.cypress.io/guides/references/configuration#excludeSpecPattern)
+
+## External adapters
+
+As mentioned in [Creating an Adapter](https://gatsbyjs.com/docs/how-to/previews-deploys-hosting/creating-an-adapter/#testing) you can use this test suite for your own adapter.
+
+Copy the whole `adapters` folder, and follow [adding a new adapter](#adding-a-new-adapter).

e2e-tests/adapters/constants.ts

@@ -0,0 +1,2 @@
+export const title = "Adapters"
+export const siteDescription = "End-to-End tests for Gatsby Adapters"

e2e-tests/adapters/cypress.config.ts

@@ -0,0 +1,11 @@
+import { defineConfig } from "cypress"
+
+export default defineConfig({
+  e2e: {
+    baseUrl: `http://localhost:9000`,
+    projectId: `4enh4m`,
+    videoUploadOnPasses: false,
+    experimentalRunAllSpecs: true,
+    retries: 2,
+  },
+})

e2e-tests/adapters/cypress/configs/netlify.ts

@@ -0,0 +1,13 @@
+import { defineConfig } from "cypress"
+
+export default defineConfig({
+  e2e: {
+    baseUrl: `http://localhost:8888`,
+    // Netlify doesn't handle trailing slash behaviors really, so no use in testing it
+    excludeSpecPattern: [`cypress/e2e/trailing-slash.cy.ts`,],
+    projectId: `4enh4m`,
+    videoUploadOnPasses: false,
+    experimentalRunAllSpecs: true,
+    retries: 2,
+  },
+})

e2e-tests/adapters/cypress/e2e/basics.cy.ts

@@ -0,0 +1,44 @@
+import { title } from "../../constants"
+
+describe('Basics', () => {
+  beforeEach(() => {
+    cy.intercept("/gatsby-icon.png").as("static-folder-image")
+    cy.intercept("/static/astro-**.png").as("img-import")
+
+    cy.visit('/').waitForRouteChange()
+  })
+
+  it('should display index page', () => {
+    cy.get('h1').should('have.text', title)
+    cy.title().should('eq', 'Adapters E2E')
+  })
+  // If this test fails, run "gatsby build" and retry
+  it('should serve assets from "static" folder', () => {
+    cy.wait("@static-folder-image").should(req => {
+      expect(req.response.statusCode).to.be.gte(200).and.lt(400)
+    })
+
+    cy.get('[alt="Gatsby Monogram Logo"]').should('be.visible')
+  })
+  it('should serve assets imported through webpack', () => {
+    cy.wait("@img-import").should(req => {
+      expect(req.response.statusCode).to.be.gte(200).and.lt(400)
+    })
+
+    cy.get('[alt="Gatsby Astronaut"]').should('be.visible')
+  })
+  it(`should show custom 404 page on invalid URL`, () => {
+    cy.visit(`/non-existent-page`, {
+      failOnStatusCode: false,
+    })
+
+    cy.get('h1').should('have.text', 'Page not found')
+  })
+  it('should apply CSS', () => {
+    cy.get(`h1`).should(
+      `have.css`,
+      `color`,
+      `rgb(21, 21, 22)`
+    )
+  })
+})

e2e-tests/adapters/cypress/e2e/client-only.cy.ts

@@ -0,0 +1,89 @@
+Cypress.on('uncaught:exception', (err) => {
+  if (err.message.includes('Minified React error')) {
+    return false
+  }
+})
+
+describe('Sub-Router', () => {
+  const routes = [
+    {
+      path: "/routes/sub-router",
+      marker: "index",
+      label: "Index route"
+    },
+    {
+      path: `/routes/sub-router/page/profile`,
+      marker: `profile`,
+      label: `Dynamic route`,
+    },
+    {
+      path: `/routes/sub-router/not-found`,
+      marker: `NotFound`,
+      label: `Default route (not found)`,
+    },
+    {
+      path: `/routes/sub-router/nested`,
+      marker: `nested-page/index`,
+      label: `Index route inside nested router`,
+    },
+    {
+      path: `/routes/sub-router/nested/foo`,
+      marker: `nested-page/foo`,
+      label: `Dynamic route inside nested router`,
+    },
+    {
+      path: `/routes/sub-router/static`,
+      marker: `static-sibling`,
+      label: `Static route that is a sibling to client only path`,
+    },
+  ] as const
+
+  routes.forEach(({ path, marker, label }) => {
+    it(label, () => {
+      cy.visit(path).waitForRouteChange()
+      cy.get(`[data-testid="dom-marker"]`).contains(marker)
+
+      cy.url().should(
+        `match`,
+        new RegExp(`^${Cypress.config().baseUrl + path}/?$`)
+      )
+    })
+  })
+})
+
+describe('Paths', () => {
+  const routes = [
+    {
+      name: 'client-only',
+      param: 'dune',
+    },
+    {
+      name: 'client-only/wildcard',
+      param: 'atreides/harkonnen',
+    },
+    {
+      name: 'client-only/named-wildcard',
+      param: 'corinno/fenring',
+    },
+  ] as const
+
+  for (const route of routes) {
+    it(`should return "${route.name}" result`, () => {
+      cy.visit(`/routes/${route.name}${route.param ? `/${route.param}` : ''}`).waitForRouteChange()
+      cy.get("[data-testid=title]").should("have.text", route.name)
+      cy.get("[data-testid=params]").should("have.text", route.param)
+    })
+  }
+})
+
+describe('Prioritize', () => {
+  it('should prioritize static page over matchPath page with wildcard', () => {
+    cy.visit('/routes/client-only/prioritize').waitForRouteChange()
+    cy.get("[data-testid=title]").should("have.text", "client-only/prioritize static")
+  })
+  it('should return result for wildcard on nested prioritized path', () => {
+    cy.visit('/routes/client-only/prioritize/nested').waitForRouteChange()
+    cy.get("[data-testid=title]").should("have.text", "client-only/prioritize matchpath")
+    cy.get("[data-testid=params]").should("have.text", "nested")
+  })
+})

e2e-tests/adapters/cypress/e2e/dsg.cy.ts

@@ -0,0 +1,14 @@
+import { title } from "../../constants"
+
+describe("Deferred Static Generation (DSG)", () => {
+  it("should work correctly", () => {
+    cy.visit("/routes/dsg/static").waitForRouteChange()
+
+    cy.get("h1").contains("DSG")
+  })
+  it("should work with page queries", () => {
+    cy.visit("/routes/dsg/graphql-query").waitForRouteChange()
+
+    cy.get(`[data-testid="title"]`).should("have.text", title)
+  })
+})

e2e-tests/adapters/cypress/e2e/functions.cy.ts

@@ -0,0 +1,27 @@
+const routes = [
+  {
+    name: 'static',
+    param: '',
+  },
+  {
+    name: 'param',
+    param: 'dune',
+  },
+  {
+    name: 'wildcard',
+    param: 'atreides/harkonnen'
+  },
+  {
+    name: 'named-wildcard',
+    param: 'corinno/fenring'
+  }
+] as const
+
+describe('Functions', () => {
+  for (const route of routes) {
+    it(`should return "${route.name}" result`, () => {
+      cy.request(`/api/${route.name}${route.param ? `/${route.param}` : ''}`).as(`req-${route.name}`)
+      cy.get(`@req-${route.name}`).its('body').should('contain', `Hello World${route.param ? ` from ${route.param}` : ``}`)
+    })
+  }
+})

e2e-tests/adapters/cypress/e2e/redirects.cy.ts

@@ -0,0 +1,57 @@
+import { applyTrailingSlashOption } from "../../utils"
+
+Cypress.on("uncaught:exception", (err) => {
+  if (err.message.includes("Minified React error")) {
+    return false
+  }
+})
+
+const TRAILING_SLASH = Cypress.env(`TRAILING_SLASH`) || `never`
+
+// Those tests won't work using `gatsby serve` because it doesn't support redirects
+
+describe("Redirects", () => {
+  it("should redirect from non-existing page to existing", () => {
+    cy.visit(applyTrailingSlashOption(`/redirect`, TRAILING_SLASH), {
+      failOnStatusCode: false,
+    }).waitForRouteChange()
+      .assertRoute(applyTrailingSlashOption(`/routes/redirect/hit`, TRAILING_SLASH))
+
+    cy.get(`h1`).should(`have.text`, `Hit`)
+  })
+  it("should respect that pages take precedence over redirects", () => {
+    cy.visit(applyTrailingSlashOption(`/routes/redirect/existing`, TRAILING_SLASH), {
+      failOnStatusCode: false,
+    }).waitForRouteChange()
+      .assertRoute(applyTrailingSlashOption(`/routes/redirect/existing`, TRAILING_SLASH))
+
+    cy.get(`h1`).should(`have.text`, `Existing`)
+  })
+  it("should support hash parameter on direct visit", () => {
+    cy.visit(applyTrailingSlashOption(`/redirect`, TRAILING_SLASH) + `#anchor`, {
+      failOnStatusCode: false,
+    }).waitForRouteChange()
+
+    cy.location(`pathname`).should(`equal`, applyTrailingSlashOption(`/routes/redirect/hit`, TRAILING_SLASH))
+    cy.location(`hash`).should(`equal`, `#anchor`)
+    cy.location(`search`).should(`equal`, ``)
+  })
+  it("should support search parameter on direct visit", () => {
+    cy.visit(applyTrailingSlashOption(`/redirect`, TRAILING_SLASH) + `?query_param=hello`, {
+      failOnStatusCode: false,
+    }).waitForRouteChange()
+
+    cy.location(`pathname`).should(`equal`, applyTrailingSlashOption(`/routes/redirect/hit`, TRAILING_SLASH))
+    cy.location(`hash`).should(`equal`, ``)
+    cy.location(`search`).should(`equal`, `?query_param=hello`)
+  })
+  it("should support search & hash parameter on direct visit", () => {
+    cy.visit(applyTrailingSlashOption(`/redirect`, TRAILING_SLASH) + `?query_param=hello#anchor`, {
+      failOnStatusCode: false,
+    }).waitForRouteChange()
+
+    cy.location(`pathname`).should(`equal`, applyTrailingSlashOption(`/routes/redirect/hit`, TRAILING_SLASH))
+    cy.location(`hash`).should(`equal`, `#anchor`)
+    cy.location(`search`).should(`equal`, `?query_param=hello`)
+  })
+})

e2e-tests/adapters/cypress/e2e/slices.cy.ts

@@ -0,0 +1,9 @@
+import { siteDescription } from "../../constants"
+
+describe("Slices", () => {
+  it("should work correctly", () => {
+    cy.visit('/').waitForRouteChange()
+
+    cy.get(`footer`).should("have.text", siteDescription)
+  })
+})

e2e-tests/adapters/cypress/e2e/ssr.cy.ts

@@ -0,0 +1,39 @@
+const staticPath = "/routes/ssr/static"
+const paramPath = "/routes/ssr/param"
+
+describe("Server Side Rendering (SSR)", () => {
+  it(`direct visit no query params (${staticPath})`, () => {
+    cy.visit(staticPath).waitForRouteChange()
+    cy.get(`[data-testid="query"]`).contains(`{}`)
+    cy.get(`[data-testid="params"]`).contains(`{}`)
+  })
+
+  it(`direct visit with query params (${staticPath})`, () => {
+    cy.visit(staticPath + `?foo=bar`).waitForRouteChange()
+    cy.get(`[data-testid="query"]`).contains(`{"foo":"bar"}`)
+    cy.get(`[data-testid="params"]`).contains(`{}`)
+  })
+
+  it(`direct visit no query params (${paramPath})`, () => {
+    cy.visit(paramPath + `/foo`).waitForRouteChange()
+    cy.get(`[data-testid="query"]`).contains(`{}`)
+    cy.get(`[data-testid="params"]`).contains(`{"param":"foo"}`)
+  })
+
+  it(`direct visit with query params (${paramPath})`, () => {
+    cy.visit(paramPath + `/foo` + `?foo=bar`).waitForRouteChange()
+    cy.get(`[data-testid="query"]`).contains(`{"foo":"bar"}`)
+    cy.get(`[data-testid="params"]`).contains(`{"param":"foo"}`)
+  })
+
+  it(`should display custom 500 page`, () => {
+    const errorPath = `/routes/ssr/error-path`
+
+    cy.visit(errorPath, { failOnStatusCode: false }).waitForRouteChange()
+
+    cy.location(`pathname`)
+      .should(`equal`, errorPath)
+      .get(`h1`)
+      .should(`have.text`, `INTERNAL SERVER ERROR`)
+  })
+})