Skip to content

Commit dc0fed2

Browse files
ejose19cspotcode
andauthored
feat: add REPL top level await support (#1383)
* feat: add repl top level await support * refactor: add more files as dist-raw, adjust primordial imports * refactor: add flag support for experimental repl await * refactor: add support for Typescript input in experimental repl await * refactor: conditionally await eval result * chore: update node-repl-await file * refactor: dynamically exclude TLA diagnostics when --experimental-repl-await is set * refactor: exclude sourceMap when transpiling for experimentalReplAwait * refactor: add acorn & acorn-walk as dependencies and remove them from dist-raw * refactor: allow setting experimentalReplAwait via env & tsconfig * refactor: adjust evalCode signature to avoid being a breaking change * refactor: improve top level await mechanism * refactor: adjust ignored diagnostic codes related to top level await/for await * refactor: use evalCodeInternal and revert evalCode to previous signature * refactor: await evalAndExitOnTsError result during bin `main` * refactor: adjust node-primordials * chore: remove unused require * refactor: revert to previous implementation of node-primordials, add missing methods * refactor: adjust node-repl-await to use compatible syntax up to node 12 * fix: typo in node-repl-await * refactor: add unhandledRejection listener * chore: remove node-primordials dts * fix: typo in node version comparison * test: add top level await tests * test: fix tla test * test: add upstream test suite of tla * feat: allow REPL to be configured on start * refactor: return repl server on `start` * refactor: use a different context when useGlobal = false * test: adjust upstream tests to latest changes * refactor: adjust new line placement on top level await processing * refactor: adjust ignored codes related to TLA * test: adjust TLA tests * refactor: override target when experimental repl await is set * test: adjust tla test * test: adjust tla tests * test: adjust tla tests * test: move tla upstream tests to a separate file * refactor: lazy load processTopLevelAwait * refactor: correctly handle errors for async eval result * refactor: throw error if target is not compatible with experimental repl await * refactor: adjust main call in bin * refactor: don't exclude tla diagnostic codes when mode is "entrypoint" * refactor: move new repl start implementation to startInternal * fix: typo in config * test: adjust tla tests * test: normalize object usage in commandline * test: move upstream tla deps from testlib to its file * test: fix formatObjectCommandLine * refactor: adjust type assertion * refactor: adjust _eval to iterate changes using a for loop * refactor: adjust execution implementation in nodeEval * test: add tla test * refactor: fix processTopLevelAwait return type * refactor: restore `main` to sync, implement callback mechanism * refactor: small adjustments in repl * refactor: remove TLA support from [stdin] & [eval] * refactor: adjust tla tests * refactor: improve code reuse in repl tests * Add raw/node-repl-await.js for easier diffing in the future * Rename flag to --no-experimental-repl-await to match node; enable by default when target is high enough; avoid `process.exit()` in `create()`; change `await` detection heuristic to match node's * Minimize changes to bin, since the only async possibility is in REPL, so bin.ts does not need to handle its errors * Integrate with latest `main` branch * fix test * fix tests * fix * fix * fix tests * fix test; make test-local fix linting errors instead of blocking on them * fix * normalize paths passed to diagnosticFilters to hopefully fix windows tests * force repl's virtual file to be a module, which is safe as long as node's runtime effectively does the same thing * remove extraneous prop deletion in show-config output * remove some todos * Update src/repl.ts Co-authored-by: ejose19 <[email protected]> * refactor: move forceToBeModule from createRepl to startRepl * test: adjust tests * test: set static target for TLA tests * refactor: show hint regarding tla errors when shouldReplAwait=false * test: small adjustments * Final cleanup * increase wait time in repl test to reduce flakiness Co-authored-by: Andrew Bradley <[email protected]>
1 parent e8a4d76 commit dc0fed2

13 files changed

+1546
-178
lines changed

Diff for: dist-raw/node-primordials.js

+10
Original file line numberDiff line numberDiff line change
@@ -1,24 +1,34 @@
11
module.exports = {
2+
ArrayFrom: Array.from,
23
ArrayIsArray: Array.isArray,
34
ArrayPrototypeJoin: (obj, separator) => Array.prototype.join.call(obj, separator),
45
ArrayPrototypeShift: (obj) => Array.prototype.shift.call(obj),
56
ArrayPrototypeForEach: (arr, ...rest) => Array.prototype.forEach.apply(arr, rest),
7+
ArrayPrototypeIncludes: (arr, ...rest) => Array.prototype.includes.apply(arr, rest),
8+
ArrayPrototypeJoin: (arr, ...rest) => Array.prototype.join.apply(arr, rest),
9+
ArrayPrototypePop: (arr, ...rest) => Array.prototype.pop.apply(arr, rest),
10+
ArrayPrototypePush: (arr, ...rest) => Array.prototype.push.apply(arr, rest),
11+
FunctionPrototype: Function.prototype,
612
JSONParse: JSON.parse,
713
JSONStringify: JSON.stringify,
814
ObjectFreeze: Object.freeze,
15+
ObjectKeys: Object.keys,
916
ObjectGetOwnPropertyNames: Object.getOwnPropertyNames,
1017
ObjectDefineProperty: Object.defineProperty,
1118
ObjectPrototypeHasOwnProperty: (obj, prop) => Object.prototype.hasOwnProperty.call(obj, prop),
1219
RegExpPrototypeTest: (obj, string) => RegExp.prototype.test.call(obj, string),
20+
RegExpPrototypeSymbolReplace: (obj, ...rest) => RegExp.prototype[Symbol.replace].apply(obj, rest),
1321
SafeMap: Map,
1422
SafeSet: Set,
1523
StringPrototypeEndsWith: (str, ...rest) => String.prototype.endsWith.apply(str, rest),
1624
StringPrototypeIncludes: (str, ...rest) => String.prototype.includes.apply(str, rest),
1725
StringPrototypeLastIndexOf: (str, ...rest) => String.prototype.lastIndexOf.apply(str, rest),
1826
StringPrototypeIndexOf: (str, ...rest) => String.prototype.indexOf.apply(str, rest),
27+
StringPrototypeRepeat: (str, ...rest) => String.prototype.repeat.apply(str, rest),
1928
StringPrototypeReplace: (str, ...rest) => String.prototype.replace.apply(str, rest),
2029
StringPrototypeSlice: (str, ...rest) => String.prototype.slice.apply(str, rest),
2130
StringPrototypeSplit: (str, ...rest) => String.prototype.split.apply(str, rest),
2231
StringPrototypeStartsWith: (str, ...rest) => String.prototype.startsWith.apply(str, rest),
2332
StringPrototypeSubstr: (str, ...rest) => String.prototype.substr.apply(str, rest),
33+
SyntaxError: SyntaxError
2434
};

Diff for: dist-raw/node-repl-await.js

+254
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,254 @@
1+
// copied from https://github.com/nodejs/node/blob/88799930794045795e8abac874730f9eba7e2300/lib/internal/repl/await.js
2+
'use strict';
3+
4+
const {
5+
ArrayFrom,
6+
ArrayPrototypeForEach,
7+
ArrayPrototypeIncludes,
8+
ArrayPrototypeJoin,
9+
ArrayPrototypePop,
10+
ArrayPrototypePush,
11+
FunctionPrototype,
12+
ObjectKeys,
13+
RegExpPrototypeSymbolReplace,
14+
StringPrototypeEndsWith,
15+
StringPrototypeIncludes,
16+
StringPrototypeIndexOf,
17+
StringPrototypeRepeat,
18+
StringPrototypeSplit,
19+
StringPrototypeStartsWith,
20+
SyntaxError,
21+
} = require('./node-primordials');
22+
23+
const parser = require('acorn').Parser;
24+
const walk = require('acorn-walk');
25+
const { Recoverable } = require('repl');
26+
27+
function isTopLevelDeclaration(state) {
28+
return state.ancestors[state.ancestors.length - 2] === state.body;
29+
}
30+
31+
const noop = FunctionPrototype;
32+
const visitorsWithoutAncestors = {
33+
ClassDeclaration(node, state, c) {
34+
if (isTopLevelDeclaration(state)) {
35+
state.prepend(node, `${node.id.name}=`);
36+
ArrayPrototypePush(
37+
state.hoistedDeclarationStatements,
38+
`let ${node.id.name}; `
39+
);
40+
}
41+
42+
walk.base.ClassDeclaration(node, state, c);
43+
},
44+
ForOfStatement(node, state, c) {
45+
if (node.await === true) {
46+
state.containsAwait = true;
47+
}
48+
walk.base.ForOfStatement(node, state, c);
49+
},
50+
FunctionDeclaration(node, state, c) {
51+
state.prepend(node, `${node.id.name}=`);
52+
ArrayPrototypePush(
53+
state.hoistedDeclarationStatements,
54+
`var ${node.id.name}; `
55+
);
56+
},
57+
FunctionExpression: noop,
58+
ArrowFunctionExpression: noop,
59+
MethodDefinition: noop,
60+
AwaitExpression(node, state, c) {
61+
state.containsAwait = true;
62+
walk.base.AwaitExpression(node, state, c);
63+
},
64+
ReturnStatement(node, state, c) {
65+
state.containsReturn = true;
66+
walk.base.ReturnStatement(node, state, c);
67+
},
68+
VariableDeclaration(node, state, c) {
69+
const variableKind = node.kind;
70+
const isIterableForDeclaration = ArrayPrototypeIncludes(
71+
['ForOfStatement', 'ForInStatement'],
72+
state.ancestors[state.ancestors.length - 2].type
73+
);
74+
75+
if (variableKind === 'var' || isTopLevelDeclaration(state)) {
76+
state.replace(
77+
node.start,
78+
node.start + variableKind.length + (isIterableForDeclaration ? 1 : 0),
79+
variableKind === 'var' && isIterableForDeclaration ?
80+
'' :
81+
'void' + (node.declarations.length === 1 ? '' : ' (')
82+
);
83+
84+
if (!isIterableForDeclaration) {
85+
ArrayPrototypeForEach(node.declarations, (decl) => {
86+
state.prepend(decl, '(');
87+
state.append(decl, decl.init ? ')' : '=undefined)');
88+
});
89+
90+
if (node.declarations.length !== 1) {
91+
state.append(node.declarations[node.declarations.length - 1], ')');
92+
}
93+
}
94+
95+
const variableIdentifiersToHoist = [
96+
['var', []],
97+
['let', []],
98+
];
99+
function registerVariableDeclarationIdentifiers(node) {
100+
switch (node.type) {
101+
case 'Identifier':
102+
ArrayPrototypePush(
103+
variableIdentifiersToHoist[variableKind === 'var' ? 0 : 1][1],
104+
node.name
105+
);
106+
break;
107+
case 'ObjectPattern':
108+
ArrayPrototypeForEach(node.properties, (property) => {
109+
registerVariableDeclarationIdentifiers(property.value);
110+
});
111+
break;
112+
case 'ArrayPattern':
113+
ArrayPrototypeForEach(node.elements, (element) => {
114+
registerVariableDeclarationIdentifiers(element);
115+
});
116+
break;
117+
}
118+
}
119+
120+
ArrayPrototypeForEach(node.declarations, (decl) => {
121+
registerVariableDeclarationIdentifiers(decl.id);
122+
});
123+
124+
ArrayPrototypeForEach(
125+
variableIdentifiersToHoist,
126+
({ 0: kind, 1: identifiers }) => {
127+
if (identifiers.length > 0) {
128+
ArrayPrototypePush(
129+
state.hoistedDeclarationStatements,
130+
`${kind} ${ArrayPrototypeJoin(identifiers, ', ')}; `
131+
);
132+
}
133+
}
134+
);
135+
}
136+
137+
walk.base.VariableDeclaration(node, state, c);
138+
}
139+
};
140+
141+
const visitors = {};
142+
for (const nodeType of ObjectKeys(walk.base)) {
143+
const callback = visitorsWithoutAncestors[nodeType] || walk.base[nodeType];
144+
visitors[nodeType] = (node, state, c) => {
145+
const isNew = node !== state.ancestors[state.ancestors.length - 1];
146+
if (isNew) {
147+
ArrayPrototypePush(state.ancestors, node);
148+
}
149+
callback(node, state, c);
150+
if (isNew) {
151+
ArrayPrototypePop(state.ancestors);
152+
}
153+
};
154+
}
155+
156+
function processTopLevelAwait(src) {
157+
const wrapPrefix = '(async () => { ';
158+
const wrapped = `${wrapPrefix}${src} })()`;
159+
const wrappedArray = ArrayFrom(wrapped);
160+
let root;
161+
try {
162+
root = parser.parse(wrapped, { ecmaVersion: 'latest' });
163+
} catch (e) {
164+
if (StringPrototypeStartsWith(e.message, 'Unterminated '))
165+
throw new Recoverable(e);
166+
// If the parse error is before the first "await", then use the execution
167+
// error. Otherwise we must emit this parse error, making it look like a
168+
// proper syntax error.
169+
const awaitPos = StringPrototypeIndexOf(src, 'await');
170+
const errPos = e.pos - wrapPrefix.length;
171+
if (awaitPos > errPos)
172+
return null;
173+
// Convert keyword parse errors on await into their original errors when
174+
// possible.
175+
if (errPos === awaitPos + 6 &&
176+
StringPrototypeIncludes(e.message, 'Expecting Unicode escape sequence'))
177+
return null;
178+
if (errPos === awaitPos + 7 &&
179+
StringPrototypeIncludes(e.message, 'Unexpected token'))
180+
return null;
181+
const line = e.loc.line;
182+
const column = line === 1 ? e.loc.column - wrapPrefix.length : e.loc.column;
183+
let message = '\n' + StringPrototypeSplit(src, '\n')[line - 1] + '\n' +
184+
StringPrototypeRepeat(' ', column) +
185+
'^\n\n' + RegExpPrototypeSymbolReplace(/ \([^)]+\)/, e.message, '');
186+
// V8 unexpected token errors include the token string.
187+
if (StringPrototypeEndsWith(message, 'Unexpected token'))
188+
message += " '" +
189+
// Wrapper end may cause acorn to report error position after the source
190+
((src.length - 1) >= (e.pos - wrapPrefix.length)
191+
? src[e.pos - wrapPrefix.length]
192+
: src[src.length - 1]) +
193+
"'";
194+
// eslint-disable-next-line no-restricted-syntax
195+
throw new SyntaxError(message);
196+
}
197+
const body = root.body[0].expression.callee.body;
198+
const state = {
199+
body,
200+
ancestors: [],
201+
hoistedDeclarationStatements: [],
202+
replace(from, to, str) {
203+
for (let i = from; i < to; i++) {
204+
wrappedArray[i] = '';
205+
}
206+
if (from === to) str += wrappedArray[from];
207+
wrappedArray[from] = str;
208+
},
209+
prepend(node, str) {
210+
wrappedArray[node.start] = str + wrappedArray[node.start];
211+
},
212+
append(node, str) {
213+
wrappedArray[node.end - 1] += str;
214+
},
215+
containsAwait: false,
216+
containsReturn: false
217+
};
218+
219+
walk.recursive(body, state, visitors);
220+
221+
// Do not transform if
222+
// 1. False alarm: there isn't actually an await expression.
223+
// 2. There is a top-level return, which is not allowed.
224+
if (!state.containsAwait || state.containsReturn) {
225+
return null;
226+
}
227+
228+
const last = body.body[body.body.length - 1];
229+
if (last.type === 'ExpressionStatement') {
230+
// For an expression statement of the form
231+
// ( expr ) ;
232+
// ^^^^^^^^^^ // last
233+
// ^^^^ // last.expression
234+
//
235+
// We do not want the left parenthesis before the `return` keyword;
236+
// therefore we prepend the `return (` to `last`.
237+
//
238+
// On the other hand, we do not want the right parenthesis after the
239+
// semicolon. Since there can only be more right parentheses between
240+
// last.expression.end and the semicolon, appending one more to
241+
// last.expression should be fine.
242+
state.prepend(last, 'return (');
243+
state.append(last.expression, ')');
244+
}
245+
246+
return (
247+
ArrayPrototypeJoin(state.hoistedDeclarationStatements, '') +
248+
ArrayPrototypeJoin(wrappedArray, '')
249+
);
250+
}
251+
252+
module.exports = {
253+
processTopLevelAwait
254+
};

Diff for: package-lock.json

+14-16
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Diff for: package.json

+3-1
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,7 @@
6666
"test-spec": "ava",
6767
"test-cov": "nyc ava",
6868
"test": "npm run build && npm run lint && npm run test-cov --",
69-
"test-local": "npm run lint && npm run build-tsc && npm run build-pack && npm run test-spec --",
69+
"test-local": "npm run lint-fix && npm run build-tsc && npm run build-pack && npm run test-spec --",
7070
"coverage-report": "nyc report --reporter=lcov",
7171
"prepare": "npm run clean && npm run build-nopack",
7272
"api-extractor": "api-extractor run --local --verbose"
@@ -162,6 +162,8 @@
162162
"@tsconfig/node12": "^1.0.7",
163163
"@tsconfig/node14": "^1.0.0",
164164
"@tsconfig/node16": "^1.0.2",
165+
"acorn": "^8.4.1",
166+
"acorn-walk": "^8.1.1",
165167
"arg": "^4.1.0",
166168
"create-require": "^1.1.0",
167169
"diff": "^4.0.1",

0 commit comments

Comments
 (0)