chore: add the Next.js e2e test suite

[ascorbic]

Summary

Adds most of the e2e test suite from the Next.js monorepo. This is a follow-on from #1773, which added the tooling and one suite.

The approach I took was to copy across all tests, fix any failures that were problems with the tooling or errors in the tests, disable any that aren't appropriate to us (e.g. they're testing client-side stuff, or dev mode), and then move them into one of three folders. The tests folder has suites that fully pass with no changes.

The modified-tests folder has tests that have been changed in some way - whether to disable individual tests or to fix errors in the tests. This is so we don't try to update them from upstream without checking this. I have disabled any failing tests and commented each with // NTL Fail. The skipped tests can be run by setting the env var RUN_SKIPPED_TESTS.

Finally there are disabled-tests. These are suites that we do want to pass, but currently fail all tests or almost all tests. Each of these is usually because of a single bug or incompatibility. These are all tracked in the umbrella issue.

Depending on which suites are enabled, these tests run around 30-50 site deploys. I have made changes to enable this to happen efficiently. The way I have done this is by generating a matrix of GitHub Actions jobs, with one per test file (most of which would generate one site). I use jest-junit to report the results, and these are uploaded as task artifacts. There is then a "Report" job that runs after the matrix, which combines these reports and generates a summary of all of the results. This can be seen in the action summary page.

The way that the tests are run is by re-implementing the NextDeployInstance class from the Next.js monorepo. The original would deploy to Vercel. This instead deploys to Netlify, using the Netlify CLI. It uses ntl deploy --build --json, which allows it to extract the deploy preview URL, against which it runs the tests. The tests all use the same site for the deploy previews, which does not have a linked repo.

The builds are performed in the GitHub Action rather than in Netlify. The workflow file has comments explaining how it distributes the sites across dozens of worker jobs. The test tooling generates the site from the local fixture in a tmp directory, with the Netlify Next runtime installed as a file dependency. It uses yarn for this as it handles these better. If you pass the env var SITE_URL then it will run the tests against this rather than deploying the site. This only makes sense when testing an individual site (i.e. running one suite) because each suite uses a different test site.

Test plan

1. Have a look at the test summary page. You can find it by clicking on the "Next.js e2e test suite" link in the Checks section below, and then clicking "Summary" in the sidebar.
2. There should be no need to review the tests themselves - so anything in test/e2e/*-tests

Relevant links (GitHub issues, Notion docs, etc.) or a picture of cute animal

Standard checks:

• Check the Deploy Preview's Demo site for your PR's functionality
• Add docs when necessary

---

🧪 Once merged, make sure to update the version if needed and that it was published correctly.

[netlify]

✅ Deploy Preview for netlify-plugin-nextjs-nx-monorepo-demo canceled.

Name	Link
🔨 Latest commit	cf61b0b
🔍 Latest deploy log	https://app.netlify.com/sites/netlify-plugin-nextjs-nx-monorepo-demo/deploys/637e58f97294bf00085975cd

[netlify]

✅ Deploy Preview for netlify-plugin-nextjs-static-root-demo canceled.

Name	Link
🔨 Latest commit	cf61b0b
🔍 Latest deploy log	https://app.netlify.com/sites/netlify-plugin-nextjs-static-root-demo/deploys/637e58f9fe7bb400081a3946

[netlify]

✅ Deploy Preview for netlify-plugin-nextjs-export-demo canceled.

Name	Link
🔨 Latest commit	cf61b0b
🔍 Latest deploy log	https://app.netlify.com/sites/netlify-plugin-nextjs-export-demo/deploys/637e58f9538dbe00093113de

[netlify]

✅ Deploy Preview for next-plugin-edge-middleware canceled.

Name	Link
🔨 Latest commit	cf61b0b
🔍 Latest deploy log	https://app.netlify.com/sites/next-plugin-edge-middleware/deploys/637e58f988be4d0009c6a577

[netlify]

❌ Deploy Preview for nextjs-plugin-next-11-demo failed.

Name	Link
🔨 Latest commit	d0aaab4
🔍 Latest deploy log	https://app.netlify.com/sites/nextjs-plugin-next-11-demo/deploys/637784b99a3146000958c703

[netlify]

✅ Deploy Preview for netlify-plugin-nextjs-next-auth-demo canceled.

Name	Link
🔨 Latest commit	cf61b0b
🔍 Latest deploy log	https://app.netlify.com/sites/netlify-plugin-nextjs-next-auth-demo/deploys/637e58f90d14ed0009cc18f8

[netlify]

✅ Deploy Preview for netlify-plugin-nextjs-demo canceled.

Name	Link
🔨 Latest commit	cf61b0b
🔍 Latest deploy log	https://app.netlify.com/sites/netlify-plugin-nextjs-demo/deploys/637e58f99fab2300083d18d0

[netlify]

✅ Deploy Preview for next-i18next-demo canceled.

Name	Link
🔨 Latest commit	cf61b0b
🔍 Latest deploy log	https://app.netlify.com/sites/next-i18next-demo/deploys/637e58f9ab63c90008a24bda

[netlify]

✅ Deploy Preview for nextjs-plugin-custom-routes-demo canceled.

Name	Link
🔨 Latest commit	cf61b0b
🔍 Latest deploy log	https://app.netlify.com/sites/nextjs-plugin-custom-routes-demo/deploys/637e58f9a9b7d90008e862c2

[netlify]

✅ Deploy Preview for next-plugin-canary canceled.

Name	Link
🔨 Latest commit	cf61b0b
🔍 Latest deploy log	https://app.netlify.com/sites/next-plugin-canary/deploys/637e58f9f39b430009a28f6a

[orinokai]

Looks good to me. This is a such a great improvement (also a technical marvel!). Just two questions:

1. I'm wondering how well this would scale so we can also test previous versions, e.g. Next 11 and 12?
2. Would we want to make the Report job a required pass?

[ascorbic]

1. I think we'd need specific tests for those, as these do test 13-specific features.
2. Yes, I was going to do that once this was merged