refactor(testing): fix flaky terminal test #3230
Conversation
Codecov Report
@@ Coverage Diff @@
## main #3230 +/- ##
=======================================
Coverage 46.90% 46.90%
=======================================
Files 23 23
Lines 1196 1196
Branches 237 237
=======================================
Hits 561 561
Misses 451 451
Partials 184 184 Continue to review full report at Codecov.
|
jsjoeio
force-pushed
the
jsjoeio/fix-e2e-tests
branch
from
45df7b5 to
7529975
Compare
jsjoeio
force-pushed
the
jsjoeio/fix-e2e-tests
branch
from
7529975 to
449c6da
Compare
| // Read more: https://thisthat.dev/dom-content-loaded-vs-load/ | ||
| await this.page.waitForLoadState("load") | ||
| // Give it an extra second just in case it's feeling extra slow | ||
| await this.page.waitForTimeout(1000) |
There was a problem hiding this comment.
ahah these things make me sad, totally fair though
There was a problem hiding this comment.
lol i know. sadly, this timeout was the only thing that made it consistently pass.
| @@ -5,6 +5,7 @@ import { CODE_SERVER_ADDRESS } from "../../utils/constants" | |||
| // See Playwright docs: https://playwright.dev/docs/pom/ | |||
| export class CodeServer { | |||
| page: Page | |||
| editorSelector = "div.monaco-workbench" | |||
There was a problem hiding this comment.
🛠️ I realized how many places I was using this in the model so refactored it into a property of the class
| // Read more: https://thisthat.dev/dom-content-loaded-vs-load/ | ||
| await this.page.waitForLoadState("load") | ||
| // Give it an extra second just in case it's feeling extra slow | ||
| await this.page.waitForTimeout(1000) |
There was a problem hiding this comment.
lol i know. sadly, this timeout was the only thing that made it consistently pass.
| // Click text=Command Palette | ||
| await this.page.hover("text=Command Palette") | ||
| await this.page.click("text=Command Palette") | ||
|
|
||
| // Type Terminal: Focus Terminal | ||
| await this.page.keyboard.type("Terminal: Focus Terminal") | ||
|
|
||
| // Click Terminal: Focus Terminal | ||
| await this.page.hover("text=Terminal: Focus Terminal") | ||
| await this.page.click("text=Terminal: Focus Terminal") |
There was a problem hiding this comment.
This approach produces consistent results — no matter if the terminal is already focused, visible/not visible.
| // which is why we wait for it twice | ||
| await this.page.waitForSelector("div.terminal.xterm.focus") | ||
| // Wait for terminal textarea to show up | ||
| await this.page.waitForSelector("textarea.xterm-helper-textarea") |
There was a problem hiding this comment.
This also seems to be better than looking for the div.terminal.xterm.focus selector, which may or may not appear.
H/T to @kylecarbs for the suggestion 🎉
xterm source
| // It may take a second to process | ||
| await page.waitForTimeout(1000) |
There was a problem hiding this comment.
There is no great way of knowing that the terminal received both the input and the enter and processed it (because it's a <canvas> element so we can't assert the text on screen) so having this timeout accounts for any slowness that might occur.
This PR fixes a flaky implementation of the
focusTerminalmethod on theCodeServermodel.This was caught after merging into #3169
mainand the e2e tests failed here 😅Changes:
reloadUntilEditorIsVisiblemethod. It wasn't waiting for code-server to fully load before reloading. before "Editor became visible after 11 reloads -> "Editor became visible after 1 reload"focusTerminalmethod.Before
The
should show the Integrated Terminalonly passed intermittently in Firefox:✓ CodeServer should show the Integrated Terminal (20s) x 3) CodeServer should show the Integrated Terminal (47s) x 1) CodeServer should show the Integrated Terminal (47s) x 2) CodeServer should show the Integrated Terminal (47s) x 4) CodeServer should show the Integrated Terminal (48s) 1 passed (52s) 4 failed codeServer.test.ts:41:8 › [firefox] CodeServer should show the Integrated Terminal ============= codeServer.test.ts:41:8 › [firefox] CodeServer should show the Integrated Terminal ============= codeServer.test.ts:41:8 › [firefox] CodeServer should show the Integrated Terminal ============= codeServer.test.ts:41:8 › [firefox] CodeServer should show the Integrated Terminal ============= 1) codeServer.test.ts:41:8 › [firefox] CodeServer should show the Integrated Terminal ============e4bebc41-637c-4949-9b13-ddf10f64a6cc.mp4
What's happening: it's waiting for the selector
div.terminal.xterm.focusto be visible, which doesn't happen for some reason and the test times-out.After