Skip to content

fix(testing): reduce flakiness of terminal.test.ts and use 1 worker for e2e tests #3263

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Apr 30, 2021
Merged

fix(testing): reduce flakiness of terminal.test.ts and use 1 worker for e2e tests #3263

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c568300
chore: clean up yarn.lock
jsjoeio Apr 29, 2021
17f4c4c
fix(e2e): remove quotes from terminal type command
jsjoeio Apr 29, 2021
7309ea9
fix(e2e): use one worker to reduce flakiness
jsjoeio Apr 29, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions test/config.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ const config: Config = {
testDir: path.join(__dirname, "e2e"), // Search for tests in this directory.
timeout: 60000, // Each test is given 60 seconds.
retries: 3, // Retry failing tests 2 times
workers: 1,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this make tests slower?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this make tests slower?

Yes 😢 Unfortunately, it seems having these tests run in parallel against the same code-server application leads to flakiness (i.e. weird things happen). It's probably the way code-server is built and/or having concurrent connections to the same instance.

More about workers:

Each worker process creates a new environment to run tests. Different environments always run in different workers. By default, Folio reuses the worker as much as it can to make testing faster, but it will create a new worker when retrying tests, after any test failure, to initialize a new environment, or just to speed up test execution if the worker limit is not reached.
— folio docs (playwright-test uses folio under the hood)

}

if (process.env.CI) {
Expand Down
2 changes: 1 addition & 1 deletion test/e2e/terminal.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ test.describe("Integrated Terminal", () => {
await codeServer.focusTerminal()

await page.waitForLoadState("load")
await page.keyboard.type(`echo '${testString}' > '${tmpFile}'`)
await page.keyboard.type(`echo ${testString} > ${tmpFile}`)
await page.keyboard.press("Enter")
// It may take a second to process
await page.waitForTimeout(1000)
Expand Down
5 changes: 0 additions & 5 deletions yarn.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3141,11 +3141,6 @@ eslint-plugin-import@^2.18.2:
resolve "^1.17.0"
tsconfig-paths "^3.9.0"

eslint-plugin-jest-playwright@^0.2.1:
version "0.2.1"
resolved "https://registry.yarnpkg.com/eslint-plugin-jest-playwright/-/eslint-plugin-jest-playwright-0.2.1.tgz#8778fee9d5915132a03d94370d3eea0a7ddd08f3"
integrity sha512-BicKUJUpVPsLbHN8c5hYaZn6pv8PCMjBGHXUfvlY1p75fh4APVfX2gTK14HuiR8/Bv3fKBQu5MTaqCro4E3OHg==

eslint-plugin-prettier@^3.1.0:
version "3.4.0"
resolved "https://registry.yarnpkg.com/eslint-plugin-prettier/-/eslint-plugin-prettier-3.4.0.tgz#cdbad3bf1dbd2b177e9825737fe63b476a08f0c7"
Expand Down