Skip to content

Commit ada73f9

Browse files
evanwevanw
committed
fix #1647: add a fallback for "npm --no-optional"
1 parent 1152047 commit ada73f9

File tree

5 files changed

+265
-71
lines changed

5 files changed

+265
-71
lines changed

‎CHANGELOG.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,14 @@
1212

1313
With this release, the optimization is now implemented with a [link](https://www.man7.org/linux/man-pages/man2/link.2.html) operation followed by a [rename](https://www.man7.org/linux/man-pages/man2/rename.2.html) operation. This should always leave the package in a working state even if either step fails.
1414

15+
* Add a fallback for `npm install esbuild --no-optional` ([#1647](https://github.com/evanw/esbuild/issues/1647))
16+
17+
The installation method for esbuild's platform-specific binary executable was recently changed in version 0.13.0. Before that version esbuild downloaded it in an install script, and after that version esbuild lets the package manager download it using the `optionalDependencies` feature in `package.json`. This change was made because downloading the binary executable in an install script never really fully worked. The reasons are complex but basically there are a variety of edge cases where people people want to install esbuild in environments that they have customized such that downloading esbuild isn't possible. Using `optionalDependencies` instead lets the package manager deal with it instead, which should work fine in all cases (either that or your package manager has a bug, but that's not esbuild's problem).
18+
19+
There is one case where this new installation method doesn't work: if you pass the `--no-optional` flag to npm to disable the `optionalDependencies` feature. If you do this, you prevent esbuild from being installed. This is not a problem with esbuild because you are manually enabling a flag to change npm's behavior such that esbuild doesn't install correctly. However, people still want to do this.
20+
21+
With this release, esbuild will now fall back to the old installation method if the new installation method fails. **THIS MAY NOT WORK.** The new `optionalDependencies` installation method is the only supported way to install esbuild with npm. The old downloading installation method was removed because it doesn't always work. The downloading method is only being provided to try to be helpful but it's not the supported installation method. If you pass `--no-optional` and the download fails due to some environment customization you did, the recommended fix is to just remove the `--no-optional` flag.
22+
1523
* Support the new `.mts` and `.cts` TypeScript file extensions
1624

1725
The upcoming version 4.5 of TypeScript has two new file extensions: `.mts` and `.cts`. Files with these extensions can be imported using the `.mjs` and `.cjs`, respectively. So the statement `import "./foo.mjs"` in TypeScript can actually succeed even if the file `./foo.mjs` doesn't exist on the file system as long as the file `./foo.mts` does exist. The import path with the `.mjs` extension is automatically re-routed to the corresponding file with the `.mts` extension at type-checking time by the TypeScript compiler. See [the TypeScript 4.5 beta announcement](https://devblogs.microsoft.com/typescript/announcing-typescript-4-5-beta/#new-file-extensions) for details.

‎lib/npm/node-install.ts

Lines changed: 209 additions & 43 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,10 @@
1-
import { binPathForCurrentPlatform } from './node-platform';
1+
import { downloadedBinPath, pkgAndSubpathForCurrentPlatform } from './node-platform';
22

33
import fs = require('fs');
44
import os = require('os');
55
import path = require('path');
6+
import zlib = require('zlib');
7+
import https = require('https');
68
import child_process = require('child_process');
79

810
declare const ESBUILD_VERSION: string;
@@ -28,12 +30,104 @@ function isYarn2OrAbove(): boolean {
2830
return false;
2931
}
3032

31-
// This feature was added to give external code a way to modify the binary
32-
// path without modifying the code itself. Do not remove this because
33-
// external code relies on this (in addition to esbuild's own test suite).
34-
if (process.env.ESBUILD_BINARY_PATH) {
33+
function fetch(url: string): Promise<Buffer> {
34+
return new Promise((resolve, reject) => {
35+
https.get(url, res => {
36+
if ((res.statusCode === 301 || res.statusCode === 302) && res.headers.location)
37+
return fetch(res.headers.location).then(resolve, reject);
38+
if (res.statusCode !== 200)
39+
return reject(new Error(`Server responded with ${res.statusCode}`));
40+
let chunks: Buffer[] = [];
41+
res.on('data', chunk => chunks.push(chunk));
42+
res.on('end', () => resolve(Buffer.concat(chunks)));
43+
}).on('error', reject);
44+
});
45+
}
46+
47+
function extractFileFromTarGzip(buffer: Buffer, subpath: string): Buffer {
48+
try {
49+
buffer = zlib.unzipSync(buffer);
50+
} catch (err: any) {
51+
throw new Error(`Invalid gzip data in archive: ${err && err.message || err}`);
52+
}
53+
let str = (i: number, n: number) => String.fromCharCode(...buffer.subarray(i, i + n)).replace(/\0.*$/, '');
54+
let offset = 0;
55+
subpath = `package/${subpath}`;
56+
while (offset < buffer.length) {
57+
let name = str(offset, 100);
58+
let size = parseInt(str(offset + 124, 12), 8);
59+
offset += 512;
60+
if (!isNaN(size)) {
61+
if (name === subpath) return buffer.subarray(offset, offset + size);
62+
offset += (size + 511) & ~511;
63+
}
64+
}
65+
throw new Error(`Could not find ${JSON.stringify(subpath)} in archive`);
66+
}
67+
68+
function installUsingNPM(pkg: string, subpath: string, binPath: string): void {
69+
// Erase "npm_config_global" so that "npm install --global esbuild" works.
70+
// Otherwise this nested "npm install" will also be global, and the install
71+
// will deadlock waiting for the global installation lock.
72+
const env = { ...process.env, npm_config_global: undefined };
73+
74+
// Create a temporary directory inside the "esbuild" package with an empty
75+
// "package.json" file. We'll use this to run "npm install" in.
76+
const esbuildLibDir = path.dirname(require.resolve('esbuild'));
77+
const installDir = path.join(esbuildLibDir, 'npm-install');
78+
fs.mkdirSync(installDir);
79+
try {
80+
fs.writeFileSync(path.join(installDir, 'package.json'), '{}');
81+
82+
// Run "npm install" in the temporary directory which should download the
83+
// desired package. Try to avoid unnecessary log output. This uses the "npm"
84+
// command instead of a HTTP request so that it hopefully works in situations
85+
// where HTTP requests are blocked but the "npm" command still works due to,
86+
// for example, a custom configured npm registry and special firewall rules.
87+
child_process.execSync(`npm install --loglevel=error --prefer-offline --no-audit --progress=false ${pkg}@${ESBUILD_VERSION}`,
88+
{ cwd: installDir, stdio: 'pipe', env });
89+
90+
// Move the downloaded binary executable into place. The destination path
91+
// is the same one that the JavaScript API code uses so it will be able to
92+
// find the binary executable here later.
93+
const installedBinPath = path.join(installDir, 'node_modules', pkg, subpath);
94+
fs.renameSync(installedBinPath, binPath);
95+
} finally {
96+
// Try to clean up afterward so we don't unnecessarily waste file system
97+
// space. Leaving nested "node_modules" directories can also be problematic
98+
// for certain tools that scan over the file tree and expect it to have a
99+
// certain structure.
100+
try {
101+
removeRecursive(installDir);
102+
} catch {
103+
// Removing a file or directory can randomly break on Windows, returning
104+
// EBUSY for an arbitrary length of time. I think this happens when some
105+
// other program has that file or directory open (e.g. an anti-virus
106+
// program). This is fine on Unix because the OS just unlinks the entry
107+
// but keeps the reference around until it's unused. There's nothing we
108+
// can do in this case so we just leave the directory there.
109+
}
110+
}
111+
}
112+
113+
function removeRecursive(dir: string): void {
114+
for (const entry of fs.readdirSync(dir)) {
115+
const entryPath = path.join(dir, entry);
116+
let stats;
117+
try {
118+
stats = fs.lstatSync(entryPath);
119+
} catch {
120+
continue; // Guard against https://github.com/nodejs/node/issues/4760
121+
}
122+
if (stats.isDirectory()) removeRecursive(entryPath);
123+
else fs.unlinkSync(entryPath);
124+
}
125+
fs.rmdirSync(dir);
126+
}
127+
128+
function applyManualBinaryPathOverride(overridePath: string): void {
35129
// Patch the CLI use case (the "esbuild" command)
36-
const pathString = JSON.stringify(process.env.ESBUILD_BINARY_PATH);
130+
const pathString = JSON.stringify(overridePath);
37131
fs.writeFileSync(toPath, `#!/usr/bin/env node\n` +
38132
`require('child_process').execFileSync(${pathString}, process.argv.slice(2), { stdio: 'inherit' });\n`);
39133

@@ -43,45 +137,117 @@ if (process.env.ESBUILD_BINARY_PATH) {
43137
fs.writeFileSync(libMain, `var ESBUILD_BINARY_PATH = ${pathString};\n${code}`);
44138
}
45139

46-
// This package contains a "bin/esbuild" JavaScript file that finds and runs
47-
// the appropriate binary executable. However, this means that running the
48-
// "esbuild" command runs another instance of "node" which is way slower than
49-
// just running the binary executable directly.
50-
//
51-
// Here we optimize for this by replacing the JavaScript file with the binary
52-
// executable at install time. This optimization does not work on Windows
53-
// because on Windows the binary executable must be called "esbuild.exe"
54-
// instead of "esbuild". This also doesn't work with Yarn 2+ because the Yarn
55-
// developers don't think binary modules should be used. See this thread for
56-
// details: https://github.com/yarnpkg/berry/issues/882. This optimization also
57-
// doesn't apply when npm's "--ignore-scripts" flag is used since in that case
58-
// this install script will not be run.
59-
else if (os.platform() !== 'win32' && !isYarn2OrAbove()) {
60-
const bin = binPathForCurrentPlatform();
61-
const tempPath = path.join(__dirname, 'bin-esbuild');
140+
function maybeOptimizePackage(binPath: string): void {
141+
// This package contains a "bin/esbuild" JavaScript file that finds and runs
142+
// the appropriate binary executable. However, this means that running the
143+
// "esbuild" command runs another instance of "node" which is way slower than
144+
// just running the binary executable directly.
145+
//
146+
// Here we optimize for this by replacing the JavaScript file with the binary
147+
// executable at install time. This optimization does not work on Windows
148+
// because on Windows the binary executable must be called "esbuild.exe"
149+
// instead of "esbuild". This also doesn't work with Yarn 2+ because the Yarn
150+
// developers don't think binary modules should be used. See this thread for
151+
// details: https://github.com/yarnpkg/berry/issues/882. This optimization also
152+
// doesn't apply when npm's "--ignore-scripts" flag is used since in that case
153+
// this install script will not be run.
154+
if (os.platform() !== 'win32' && !isYarn2OrAbove()) {
155+
const tempPath = path.join(__dirname, 'bin-esbuild');
156+
try {
157+
// First link the binary with a temporary file. If this fails and throws an
158+
// error, then we'll just end up doing nothing. This uses a hard link to
159+
// avoid taking up additional space on the file system.
160+
fs.linkSync(binPath, tempPath);
161+
162+
// Then use rename to atomically replace the target file with the temporary
163+
// file. If this fails and throws an error, then we'll just end up leaving
164+
// the temporary file there, which is harmless.
165+
fs.renameSync(tempPath, toPath);
166+
167+
// If we get here, then we know that the target location is now a binary
168+
// executable instead of a JavaScript file.
169+
isToPathJS = false;
170+
} catch {
171+
// Ignore errors here since this optimization is optional
172+
}
173+
}
174+
}
175+
176+
async function downloadDirectlyFromNPM(pkg: string, subpath: string, binPath: string): Promise<void> {
177+
// If that fails, the user could have npm configured incorrectly or could not
178+
// have npm installed. Try downloading directly from npm as a last resort.
179+
const url = `https://registry.npmjs.org/${pkg}/-/${pkg}-${ESBUILD_VERSION}.tgz`;
180+
console.error(`[esbuild] Trying to download ${JSON.stringify(url)}`);
62181
try {
63-
// First link the binary with a temporary file. If this fails and throws an
64-
// error, then we'll just end up doing nothing. This uses a hard link to
65-
// avoid taking up additional space on the file system.
66-
fs.linkSync(bin, tempPath);
67-
68-
// Then use rename to atomically replace the target file with the temporary
69-
// file. If this fails and throws an error, then we'll just end up leaving
70-
// the temporary file there, which is harmless.
71-
fs.renameSync(tempPath, toPath);
72-
73-
// If we get here, then we know that the target location is now a binary
74-
// executable instead of a JavaScript file.
75-
isToPathJS = false;
76-
} catch (e) {
77-
// Ignore errors here since this optimization is optional
182+
fs.writeFileSync(binPath, extractFileFromTarGzip(await fetch(url), subpath));
183+
fs.chmodSync(binPath, 0o755);
184+
} catch (e: any) {
185+
console.error(`[esbuild] Failed to download ${JSON.stringify(url)}: ${e && e.message || e}`);
186+
throw e;
78187
}
79188
}
80189

81-
if (isToPathJS) {
82-
// We need "node" before this command since it's a JavaScript file
83-
validateBinaryVersion('node', toPath);
84-
} else {
85-
// This is no longer a JavaScript file so don't run it using "node"
86-
validateBinaryVersion(toPath);
190+
async function checkAndPreparePackage(): Promise<void> {
191+
// This feature was added to give external code a way to modify the binary
192+
// path without modifying the code itself. Do not remove this because
193+
// external code relies on this (in addition to esbuild's own test suite).
194+
if (process.env.ESBUILD_BINARY_PATH) {
195+
applyManualBinaryPathOverride(process.env.ESBUILD_BINARY_PATH);
196+
return;
197+
}
198+
199+
const { pkg, subpath } = pkgAndSubpathForCurrentPlatform();
200+
201+
let binPath: string;
202+
try {
203+
// First check for the binary package from our "optionalDependencies". This
204+
// package should have been installed alongside this package at install time.
205+
binPath = require.resolve(`${pkg}/${subpath}`);
206+
} catch (e) {
207+
console.error(`[esbuild] Failed to find package "${pkg}" on the file system
208+
209+
This can happen if you use the "--no-optional" flag. The "optionalDependencies"
210+
package.json feature is used by esbuild to install the correct binary executable
211+
for your current platform. This install script will now attempt to work around
212+
this. If that fails, you need to remove the "--no-optional" flag to use esbuild.
213+
`);
214+
215+
// If that didn't work, then someone probably installed esbuild with the
216+
// "--no-optional" flag. Attempt to compensate for this by downloading the
217+
// package using a nested call to "npm" instead.
218+
//
219+
// THIS MAY NOT WORK. Package installation uses "optionalDependencies" for
220+
// a reason: manually downloading the package has a lot of obscure edge
221+
// cases that fail because people have customized their environment in
222+
// some strange way that breaks downloading. This code path is just here
223+
// to be helpful but it's not the supported way of installing esbuild.
224+
binPath = downloadedBinPath(pkg, subpath);
225+
try {
226+
console.error(`[esbuild] Trying to install package "${pkg}" using npm`);
227+
installUsingNPM(pkg, subpath, binPath);
228+
} catch (e2: any) {
229+
console.error(`[esbuild] Failed to install package "${pkg}" using npm: ${e2 && e2.message || e2}`);
230+
231+
// If that didn't also work, then something is likely wrong with the "npm"
232+
// command. Attempt to compensate for this by manually downloading the
233+
// package from the npm registry over HTTP as a last resort.
234+
try {
235+
await downloadDirectlyFromNPM(pkg, subpath, binPath);
236+
} catch (e3: any) {
237+
throw new Error(`Failed to install package "${pkg}"`);
238+
}
239+
}
240+
}
241+
242+
maybeOptimizePackage(binPath);
87243
}
244+
245+
checkAndPreparePackage().then(() => {
246+
if (isToPathJS) {
247+
// We need "node" before this command since it's a JavaScript file
248+
validateBinaryVersion('node', toPath);
249+
} else {
250+
// This is no longer a JavaScript file so don't run it using "node"
251+
validateBinaryVersion(toPath);
252+
}
253+
});

0 commit comments

Comments
 (0)