Skip to content

fix: add require shims #2050

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
May 3, 2023
Merged

fix: add require shims #2050

merged 5 commits into from
May 3, 2023

Conversation

orinokai
Copy link
Contributor

Summary

A new turbopack feature allows free variables to be aliased to modules, which causes a Next.js reference to Buffer to be bundled as require("node:buffer"), which is not supported by Deno.

This PR adds a shim for require that specifically loads Buffer for this case only.

Test plan

Incoming...

Relevant links (GitHub issues, Notion docs, etc.) or a picture of cute animal

Fixes #2047

@netlify
Copy link

netlify bot commented Apr 17, 2023

Deploy Preview for netlify-plugin-nextjs-nx-monorepo-demo ready!

Name Link
🔨 Latest commit 946533e
🔍 Latest deploy log https://app.netlify.com/sites/netlify-plugin-nextjs-nx-monorepo-demo/deploys/6451fd2a15087e0008d0875f
😎 Deploy Preview https://deploy-preview-2050--netlify-plugin-nextjs-nx-monorepo-demo.netlify.app/
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@netlify
Copy link

netlify bot commented Apr 17, 2023

Deploy Preview for netlify-plugin-nextjs-export-demo ready!

Name Link
🔨 Latest commit 946533e
🔍 Latest deploy log https://app.netlify.com/sites/netlify-plugin-nextjs-export-demo/deploys/6451fd2a9c9a540008fb6d0e
😎 Deploy Preview https://deploy-preview-2050--netlify-plugin-nextjs-export-demo.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@netlify
Copy link

netlify bot commented Apr 17, 2023

Deploy Preview for next-plugin-canary ready!

Name Link
🔨 Latest commit 946533e
🔍 Latest deploy log https://app.netlify.com/sites/next-plugin-canary/deploys/6451fd2a55ec37000872d22d
😎 Deploy Preview https://deploy-preview-2050--next-plugin-canary.netlify.app/
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@netlify
Copy link

netlify bot commented Apr 17, 2023

Deploy Preview for netlify-plugin-nextjs-next-auth-demo ready!

Name Link
🔨 Latest commit 946533e
🔍 Latest deploy log https://app.netlify.com/sites/netlify-plugin-nextjs-next-auth-demo/deploys/6451fd2a34ae7900085e569d
😎 Deploy Preview https://deploy-preview-2050--netlify-plugin-nextjs-next-auth-demo.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@netlify
Copy link

netlify bot commented Apr 17, 2023

Deploy Preview for nextjs-plugin-custom-routes-demo ready!

Name Link
🔨 Latest commit 946533e
🔍 Latest deploy log https://app.netlify.com/sites/nextjs-plugin-custom-routes-demo/deploys/6451fd2a3fb4af000891553e
😎 Deploy Preview https://deploy-preview-2050--nextjs-plugin-custom-routes-demo.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@netlify
Copy link

netlify bot commented Apr 17, 2023

Deploy Preview for netlify-plugin-nextjs-static-root-demo ready!

Name Link
🔨 Latest commit 946533e
🔍 Latest deploy log https://app.netlify.com/sites/netlify-plugin-nextjs-static-root-demo/deploys/6451fd2af86e9400075fc96d
😎 Deploy Preview https://deploy-preview-2050--netlify-plugin-nextjs-static-root-demo.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@netlify
Copy link

netlify bot commented Apr 17, 2023

Deploy Preview for next-i18next-demo ready!

Name Link
🔨 Latest commit 946533e
🔍 Latest deploy log https://app.netlify.com/sites/next-i18next-demo/deploys/6451fd2abb305600085ac8f7
😎 Deploy Preview https://deploy-preview-2050--next-i18next-demo.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@netlify
Copy link

netlify bot commented Apr 17, 2023

Deploy Preview for netlify-plugin-nextjs-demo ready!

Name Link
🔨 Latest commit 946533e
🔍 Latest deploy log https://app.netlify.com/sites/netlify-plugin-nextjs-demo/deploys/6451fd2b599ad60008162af2
😎 Deploy Preview https://deploy-preview-2050--netlify-plugin-nextjs-demo.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@netlify
Copy link

netlify bot commented Apr 17, 2023

Deploy Preview for next-plugin-edge-middleware ready!

Name Link
🔨 Latest commit 946533e
🔍 Latest deploy log https://app.netlify.com/sites/next-plugin-edge-middleware/deploys/6451fd2a9689a00008ea1d73
😎 Deploy Preview https://deploy-preview-2050--next-plugin-edge-middleware.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@github-actions github-actions bot added the type: bug code to address defects in shipped code label Apr 17, 2023
@ascorbic
Copy link
Contributor

I think adding require potentially opens up a world of pain. I don't get why it's generating a cjs require anyway. Has Deno added support for "node: imports to Deploy yet? It might be better to patch the bundle instead.

@pieh
Copy link
Contributor

pieh commented Apr 18, 2023

Just adding https://github.com/vercel/next.js/pull/47191/files#diff-be23f4ad09849e271e6c9f4f373142bc18167eb9f4fe6a86f8705b80e07995d3R202-R211 for some reference - I'm not sure of where server/sandbox is used, but require function is added there to context

Also above explains how we end up with that require (externals -> commonjs in webpack config) and also gives potential hint that if we don't want to produce code with require calls we would need to overwrite default externals config and supply alternative external for buffer (and other Node native things from above PR). If there is non-hacky way to do that I think that would be preferable to patching produced bundle after the fact? (I think you could do that in next.config.js in bundle configuration overwrites, but mutating config from build plugin/runtime is generally a mess :) )

@ascorbic
Copy link
Contributor

I think maybe this means Rob was actually right, and we should take the same approach. Put that ghastly require thing into our own preamble. Create our own NativeModuleMap and use the same approach.

@orinokai
Copy link
Contributor Author

does anyone have any insight into why Vercel pick specific properties out of the native modules? https://github.com/vercel/next.js/pull/47191/files#diff-be23f4ad09849e271e6c9f4f373142bc18167eb9f4fe6a86f8705b80e07995d3R149-R156

@nickytonline
Copy link

does anyone have any insight into why Vercel pick specific properties out of the native modules? vercel/next.js#47191 (files)

No idea why they omitted some.

@ascorbic
Copy link
Contributor

This is why! https://github.com/cloudflare/workerd/blob/main/src/node/buffer.ts

@orinokai
Copy link
Contributor Author

This is why! https://github.com/cloudflare/workerd/blob/main/src/node/buffer.ts

Thanks @ascorbic, that's really useful.

We can import the Buffer module as is, so I think this PR is ready to go. I don't think we need specific tests because the edge test currently fail without this fix at the moment.

@orinokai orinokai marked this pull request as ready for review April 26, 2023 09:45
@orinokai orinokai requested a review from a team April 26, 2023 09:45
@LekoArts LekoArts changed the title fix: add shim for require('node:buffer') fix: add shim for require('node:buffer') Apr 26, 2023
LekoArts
LekoArts previously approved these changes Apr 26, 2023
@ascorbic
Copy link
Contributor

Are you planning to add shims for the other stuff too?

if (name === 'buffer' || name === 'node:buffer') {
return { Buffer: BufferCompat }
}
throw new ReferenceError('require is not defined')
Copy link

@nickytonline nickytonline Apr 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It'll seem weird if we get the error require is not defined given that require is defined. Consdier an error message that mentions a particular module cannot be pulled in via require as that will signal to us that another shim is required.

This will help with E2E canary test runs to signal earlier on exactly what the issue is.

Something along these lines.

Suggested change
throw new ReferenceError('require is not defined')
throw new ReferenceError(`Unable to require module ${name}. A shim for module ${name} may be missing.`)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about matching the Next.js message? throw TypeError('Native module not found: ' + id)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even better 😎

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you planning to add shims for the other stuff too?

Do you mean the other stuff in the NativeModuleMap, @ascorbic? I wasn't certain whether that was specific to Vercel's infrastructure?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They're all the Node modules that are available in workerd, so which Vercel edge runtime now assumes are available. Buffer is the only on they're using in core Next.js right now (I think), but they're basically telling users that they can use them.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ascorbic okay, I'll shim all the things!

@LekoArts LekoArts dismissed their stale review April 26, 2023 13:23

stuff missing

Copy link

@nickytonline nickytonline left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great @orinokai! :shipit:

@@ -48,6 +52,26 @@ const fetch /* type {typeof globalThis.fetch} */ = async (url, init) => {
}
}

// Shim native modules that Vercel makes available
if (typeof require === 'undefined') {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So many shims. 😅

@kodiakhq kodiakhq bot merged commit e007b08 into main May 3, 2023
@kodiakhq kodiakhq bot deleted the rs/shim-require-buffer branch May 3, 2023 06:24
@nickytonline nickytonline changed the title fix: add shim for require('node:buffer') fix: add require shims May 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
automerge type: bug code to address defects in shipped code
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add missing require shims
5 participants