feat(gatsby): lazy bundle page components in dev server (#27884)

* feat(gatsby): lazy compile page components in dev server

This means we only compile the core runtime code initially — page components are only compiled when the user visits the page.

* Fix lint

* Cleanup logs

* cleanups

* fix tests

* Don't bother with navigating — if webpack doesn't update — the user probably has errors in their code or something bigger is wrong so let them just refresh, etc.

* Update packages/gatsby/src/bootstrap/requires-writer.ts

Co-authored-by: Ward Peeters <[email protected]>

* Update packages/gatsby/cache-dir/dev-loader.js

Co-authored-by: Ward Peeters <[email protected]>

* Fix lint error

* Wait until we know staticQueryHashes are written to page-data.json files

We only discover static queries for pages after the component is compiled.

Which means when we lazily add page components, we initially don't know
what static queries should be loaded. To track this, we mark page-data.json
files as not bundled & only after the staticQueryHashes is written do we
accept the page as loaded.

* Disable check in production

* Only trigger compilation/page writing once

* Also pull xhr loaded static queries for <StaticQuery>

* instance isn't defined in builds apparently

* Work around webpack hot reloading bug

* For static queries, data pushed from websocket is the most fresh

* Centralize logic and prevent it from repeatedly hitting the backend

* fix ordering of merge

* Comment and add timeout to ensure page-data writes happen after the webpack compilation

* Only add notInDevBundle during development

* Add back mistakenly removed code

* Didn't need this

* Implement @pieh's idea for telling runtime that dev bundle is loaded

* Fix tests

* Remove logs

* Put changes behind GATSBY_EXPERIMENT_LAZY_DEVJS flag

* Add temporary fork of development-runtime tests for lazy devjs PR

* Remove duplicated tests w/ cleaner method @pieh suggested

* Address @pieh feedback

* Update packages/gatsby/src/utils/start-server.ts

Co-authored-by: Michal Piechowiak <[email protected]>

* Remove global middleware

* This isn't true

* fix lint

Co-authored-by: Ward Peeters <[email protected]>
Co-authored-by: Michal Piechowiak <[email protected]>

Author: 3 people

.circleci/config.yml

@@ -117,6 +117,23 @@ aliases:
       - notify-status:
           condition: << parameters.nightly >>
 
+  e2e_tests_development_runtime_lazy_alias:
+    &e2e_tests_development_runtime_lazy_alias
+    <<: *e2e-executor
+    parameters:
+      nightly:
+        type: boolean
+        default: false
+    environment:
+      CYPRESS_PROJECT_ID: ihj5mz
+      CYPRESS_RECORD_KEY: 01acdce8-75bd-4280-9839-6cb215b2c84b
+    steps:
+      - e2e-test:
+          test_path: e2e-tests/lazy-development-runtime
+          skip_file_change_test: << parameters.nightly >>
+      - notify-status:
+          condition: << parameters.nightly >>
+
   e2e_tests_gatsby-image_alias: &e2e_tests_gatsby-image_alias
     <<: *e2e-executor
     parameters:
@@ -332,6 +349,13 @@ jobs:
   e2e_tests_development_runtime:
     <<: *e2e_tests_development_runtime_alias
 
+  e2e_tests_development_runtime_lazy_devjs:
+    <<: *e2e_tests_development_runtime_alias
+    environment:
+      GATSBY_EXPERIMENT_LAZY_DEVJS: true
+      CYPRESS_PROJECT_ID: ihj5mz
+      CYPRESS_RECORD_KEY: 01acdce8-75bd-4280-9839-6cb215b2c84b
+
   e2e_tests_development_runtime_with_experimental_react:
     <<: *e2e_tests_development_runtime_alias
 
@@ -605,6 +629,8 @@ workflows:
           <<: *e2e-test-workflow
       - e2e_tests_development_runtime:
           <<: *e2e-test-workflow
+      - e2e_tests_development_runtime_lazy_devjs:
+          <<: *e2e-test-workflow
       - e2e_tests_production_runtime:
           <<: *e2e-test-workflow
       - themes_e2e_tests_production_runtime:

packages/gatsby/cache-dir/__tests__/__snapshots__/dev-loader-lazy.js.snap

@@ -0,0 +1,20 @@
+// Jest Snapshot v1, https://goo.gl/fbAQLP
+
+exports[`Dev loader loadPage should be successful when component can be loaded 1`] = `
+Object {
+  "component": Object {
+    "chunkName": "bar",
+  },
+  "json": Object {
+    "pageContext": "something something",
+  },
+  "page": Object {
+    "componentChunkName": "chunk",
+    "matchPath": undefined,
+    "path": "/mypage/",
+    "staticQueryHashes": Array [],
+    "webpackCompilationHash": "123",
+  },
+  "staticQueryResults": Object {},
+}
+`;