vscode Review smoke test `sendKeybinding` usage

Review smoke test `sendKeybinding` usage

Open aiday-mar opened this issue 7 months ago • 3 comments

Note: in order to avoid unnecessarily breaking the build before the snow trip, it may be preferable to address this debt work starting from next week.

Last iteration, when the experimental edit context was turned on by default, it caused smoke test errors consistently. While debugging with Joao, it was found that the smoke test errors were caused by the dispatchKeybinding method which was dispatching the right arrow key press, but was not awaiting for the cursor to move right before continuing the computation. To mitigate such cases, the dispatchKeybinding was awaiting 100ms after sending the keybinding, in the hopes that the keybinding would be processed in this time frame. Since this can sometimes fail, we decided to add a callback to this method called accept which would end the dispatchKeybinding method execution when the accept method returns. In the following PR https://github.com/microsoft/vscode/pull/239902 I have added this callback and removed the 100ms timeout. I ran the VS Code CI pipeline on this PR 5 times, and it gave the following errors. Please note that the PR https://github.com/microsoft/vscode/pull/239902 is a draft closed PR and is not meant to be merged, but simply to demonstrate where the smoke tests fail.

The 5 runs were:

Run 1

https://dev.azure.com/monacotools/Monaco/_build/results?buildId=321945&view=results

linux - browser

1) VSCode Smoke Tests (Web)
       Task
         Task Quick Pick
           Tasks: Run Task
             hide property - undefined:
     Error: Timeout: get elements '.quick-input-widget .quick-input-list .monaco-list-row .quick-input-list-row > .monaco-icon-label .label-name' after 20 seconds.
      at Code.poll (/mnt/vss/_work/1/s/test/automation/out/code.js:208:23)
      at async Code.waitForElements (/mnt/vss/_work/1/s/test/automation/out/code.js:161:16)
      at async QuickInput.waitForQuickInputElements (/mnt/vss/_work/1/s/test/automation/out/quickinput.js:28:9)
      at async Task.assertTasks (/mnt/vss/_work/1/s/test/automation/out/task.js:29:13)
      at async Context.<anonymous> (out/areas/task/task-quick-pick.test.js:38:17)

macos - browser

1) VSCode Smoke Tests (Electron)
       Terminal
         Terminal stickyScroll
           should support multi-line prompt:
     Error: Failed for command sticky scroll 2, exitcode 0, text content Prompt> sticky scroll 1 
      at checkCommandAndOutput (out/areas/terminal/terminal-stickyScroll.test.js:42:19)
      at async Context.<anonymous> (out/areas/terminal/terminal-stickyScroll.test.js:58:13)

  2) VSCode Smoke Tests (Electron)
       Task
         Task Quick Pick
           Tasks: Run Task
             hide property - false:
     Error: Timeout: get text content ' .monaco-editor[data-uri$="tasks.json"] .view-lines' after 20 seconds.
      at Code.poll (/Users/runner/work/1/s/test/automation/out/code.js:208:23)
      at async Code.waitForTextContent (/Users/runner/work/1/s/test/automation/out/code.js:152:16)
      at async Editor.waitForTypeInEditor (/Users/runner/work/1/s/test/automation/out/editor.js:79:9)
      at async Task.configureTask (/Users/runner/work/1/s/test/automation/out/task.js:68:13)
      at async Context.<anonymous> (out/areas/task/task-quick-pick.test.js:33:17)

  3) VSCode Smoke Tests (Electron)
       Task
         Task Quick Pick
           Tasks: Run Task
             hide property - undefined:
     Error: Timeout: get text content ' .monaco-editor[data-uri$="tasks.json"] .view-lines' after 20 seconds.
      at Code.poll (/Users/runner/work/1/s/test/automation/out/code.js:208:23)
      at async Code.waitForTextContent (/Users/runner/work/1/s/test/automation/out/code.js:152:16)
      at async Editor.waitForTypeInEditor (/Users/runner/work/1/s/test/automation/out/editor.js:79:9)
      at async Task.configureTask (/Users/runner/work/1/s/test/automation/out/task.js:68:13)
      at async Context.<anonymous> (out/areas/task/task-quick-pick.test.js:37:17)

  4) VSCode Smoke Tests (Electron)
       Task
         Task Quick Pick
           Tasks: Run Task
             icon - icon only:
     Error: Timeout: get text content ' .monaco-editor[data-uri$="tasks.json"] .view-lines' after 20 seconds.
      at Code.poll (/Users/runner/work/1/s/test/automation/out/code.js:208:23)
      at async Code.waitForTextContent (/Users/runner/work/1/s/test/automation/out/code.js:152:16)
      at async Editor.waitForTypeInEditor (/Users/runner/work/1/s/test/automation/out/editor.js:79:9)
      at async Task.configureTask (/Users/runner/work/1/s/test/automation/out/task.js:68:13)
      at async Context.<anonymous> (out/areas/task/task-quick-pick.test.js:42:17)

Run 2

https://dev.azure.com/monacotools/Monaco/_build/results?buildId=322495&view=results

linux - browser

1) VSCode Smoke Tests (Web)
       Task
         Task Quick Pick
           Tasks: Run Task
             hide property - true:
     Error: Timeout: get text content ' .monaco-editor[data-uri$="tasks.json"] .view-lines' after 20 seconds.
      at Code.poll (/mnt/vss/_work/1/s/test/automation/out/code.js:208:23)
      at async Code.waitForTextContent (/mnt/vss/_work/1/s/test/automation/out/code.js:152:16)
      at async Editor.waitForTypeInEditor (/mnt/vss/_work/1/s/test/automation/out/editor.js:79:9)
      at async Task.configureTask (/mnt/vss/_work/1/s/test/automation/out/task.js:68:13)
      at async Context.<anonymous> (out/areas/task/task-quick-pick.test.js:29:17)

  2) VSCode Smoke Tests (Web)
       Task
         Task Quick Pick
           Tasks: Run Task
             hide property - false:
     Error: Timeout: get text content ' .monaco-editor[data-uri$="tasks.json"] .view-lines' after 20 seconds.
      at Code.poll (/mnt/vss/_work/1/s/test/automation/out/code.js:208:23)
      at async Code.waitForTextContent (/mnt/vss/_work/1/s/test/automation/out/code.js:152:16)
      at async Editor.waitForTypeInEditor (/mnt/vss/_work/1/s/test/automation/out/editor.js:79:9)
      at async Task.configureTask (/mnt/vss/_work/1/s/test/automation/out/task.js:68:13)
      at async Context.<anonymous> (out/areas/task/task-quick-pick.test.js:33:17)

macos - browser

1) VSCode Smoke Tests (Web)
       Terminal
         Terminal stickyScroll
           should support multi-line prompt:
     Error: Failed for command sticky scroll 1, exitcode 0, text content 
      at checkCommandAndOutput (out/areas/terminal/terminal-stickyScroll.test.js:42:19)
      at async Context.<anonymous> (out/areas/terminal/terminal-stickyScroll.test.js:56:13)

  2) VSCode Smoke Tests (Web)
       Task
         Task Quick Pick
           Tasks: Run Task
             hide property - false:
     Error: Timeout: get text content ' .monaco-editor[data-uri$="tasks.json"] .view-lines' after 20 seconds.
      at Code.poll (/Users/runner/work/1/s/test/automation/out/code.js:208:23)
      at async Code.waitForTextContent (/Users/runner/work/1/s/test/automation/out/code.js:152:16)
      at async Editor.waitForTypeInEditor (/Users/runner/work/1/s/test/automation/out/editor.js:79:9)
      at async Task.configureTask (/Users/runner/work/1/s/test/automation/out/task.js:68:13)
      at async Context.<anonymous> (out/areas/task/task-quick-pick.test.js:33:17)

  3) VSCode Smoke Tests (Web)
       Task
         Task Quick Pick
           Tasks: Run Task
             hide property - undefined:
     Error: Timeout: get text content ' .monaco-editor[data-uri$="tasks.json"] .view-lines' after 20 seconds.
      at Code.poll (/Users/runner/work/1/s/test/automation/out/code.js:208:23)
      at async Code.waitForTextContent (/Users/runner/work/1/s/test/automation/out/code.js:152:16)
      at async Editor.waitForTypeInEditor (/Users/runner/work/1/s/test/automation/out/editor.js:79:9)
      at async Task.configureTask (/Users/runner/work/1/s/test/automation/out/task.js:68:13)
      at async Context.<anonymous> (out/areas/task/task-quick-pick.test.js:37:17)

  4) VSCode Smoke Tests (Web)
       Task
         Task Quick Pick
           Tasks: Run Task
             icon - color only:
     Error: Timeout: get text content ' .monaco-editor[data-uri$="tasks.json"] .view-lines' after 20 seconds.
      at Code.poll (/Users/runner/work/1/s/test/automation/out/code.js:208:23)
      at async Code.waitForTextContent (/Users/runner/work/1/s/test/automation/out/code.js:152:16)
      at async Editor.waitForTypeInEditor (/Users/runner/work/1/s/test/automation/out/editor.js:79:9)
      at async Task.configureTask (/Users/runner/work/1/s/test/automation/out/task.js:68:13)
      at async Context.<anonymous> (out/areas/task/task-quick-pick.test.js:47:17)

  5) VSCode Smoke Tests (Web)
       Task
         Task Quick Pick
           Tasks: Run Task
             icon - icon & color:
     Error: Timeout: get text content ' .monaco-editor[data-uri$="tasks.json"] .view-lines' after 20 seconds.
      at Code.poll (/Users/runner/work/1/s/test/automation/out/code.js:208:23)
      at async Code.waitForTextContent (/Users/runner/work/1/s/test/automation/out/code.js:152:16)
      at async Editor.waitForTypeInEditor (/Users/runner/work/1/s/test/automation/out/editor.js:79:9)
      at async Task.configureTask (/Users/runner/work/1/s/test/automation/out/task.js:68:13)
      at async Context.<anonymous> (out/areas/task/task-quick-pick.test.js:52:17)

Run 3

https://dev.azure.com/monacotools/Monaco/_build/results?buildId=322537&view=results

linux - browser

  1) VSCode Smoke Tests (Web)
       Task
         Task Quick Pick
           Tasks: Run Task
             hide property - false:
     Error: Timeout: get text content ' .monaco-editor[data-uri$="tasks.json"] .view-lines' after 20 seconds.
      at Code.poll (/mnt/vss/_work/1/s/test/automation/out/code.js:208:23)
      at async Code.waitForTextContent (/mnt/vss/_work/1/s/test/automation/out/code.js:152:16)
      at async Editor.waitForTypeInEditor (/mnt/vss/_work/1/s/test/automation/out/editor.js:79:9)
      at async Task.configureTask (/mnt/vss/_work/1/s/test/automation/out/task.js:68:13)
      at async Context.<anonymous> (out/areas/task/task-quick-pick.test.js:33:17)

  2) VSCode Smoke Tests (Web)
       Task
         Task Quick Pick
           Tasks: Run Task
             hide property - undefined:
     Error: Timeout: get elements '.quick-input-widget .quick-input-list .monaco-list-row .quick-input-list-row > .monaco-icon-label .label-name' after 20 seconds.
      at Code.poll (/mnt/vss/_work/1/s/test/automation/out/code.js:208:23)
      at async Code.waitForElements (/mnt/vss/_work/1/s/test/automation/out/code.js:161:16)
      at async QuickInput.waitForQuickInputElements (/mnt/vss/_work/1/s/test/automation/out/quickinput.js:28:9)
      at async Task.assertTasks (/mnt/vss/_work/1/s/test/automation/out/task.js:29:13)
      at async Context.<anonymous> (out/areas/task/task-quick-pick.test.js:38:17)

macos - browser

 1) VSCode Smoke Tests (Web)
       Task
         Task Quick Pick
           Tasks: Run Task
             hide property - false:
     Error: Timeout: get text content ' .monaco-editor[data-uri$="tasks.json"] .view-lines' after 20 seconds.
      at Code.poll (/Users/runner/work/1/s/test/automation/out/code.js:208:23)
      at async Code.waitForTextContent (/Users/runner/work/1/s/test/automation/out/code.js:152:16)
      at async Editor.waitForTypeInEditor (/Users/runner/work/1/s/test/automation/out/editor.js:79:9)
      at async Task.configureTask (/Users/runner/work/1/s/test/automation/out/task.js:68:13)
      at async Context.<anonymous> (out/areas/task/task-quick-pick.test.js:33:17)

Run 4

https://dev.azure.com/monacotools/Monaco/_build/results?buildId=322579&view=results

macOS browser:

  1) VSCode Smoke Tests (Electron)
       Terminal
         Terminal stickyScroll
           should show sticky scroll when appropriate:
     Error: Failed for command sticky scroll 2, exitcode 1, text content Prompt> sticky scroll 1 
      at checkCommandAndOutput (out/areas/terminal/terminal-stickyScroll.test.js:42:19)
      at async Context.<anonymous> (out/areas/terminal/terminal-stickyScroll.test.js:52:13)

Run 5

https://dev.azure.com/monacotools/Monaco/_build/results?buildId=322773&view=results

linux browser:

  1) VSCode Smoke Tests (Web)
       Task
         Task Quick Pick
           Tasks: Run Task
             hide property - false:
     Error: Timeout: get text content ' .monaco-editor[data-uri$="tasks.json"] .view-lines' after 20 seconds.
      at Code.poll (/mnt/vss/_work/1/s/test/automation/out/code.js:208:23)
      at async Code.waitForTextContent (/mnt/vss/_work/1/s/test/automation/out/code.js:152:16)
      at async Editor.waitForTypeInEditor (/mnt/vss/_work/1/s/test/automation/out/editor.js:79:9)
      at async Task.configureTask (/mnt/vss/_work/1/s/test/automation/out/task.js:68:13)
      at async Context.<anonymous> (out/areas/task/task-quick-pick.test.js:33:17)

  2) VSCode Smoke Tests (Web)
       Task
         Task Quick Pick
           Tasks: Run Task
             hide property - undefined:
     Error: Timeout: get text content ' .monaco-editor[data-uri$="tasks.json"] .view-lines' after 20 seconds.
      at Code.poll (/mnt/vss/_work/1/s/test/automation/out/code.js:208:23)
      at async Code.waitForTextContent (/mnt/vss/_work/1/s/test/automation/out/code.js:152:16)
      at async Editor.waitForTypeInEditor (/mnt/vss/_work/1/s/test/automation/out/editor.js:79:9)
      at async Task.configureTask (/mnt/vss/_work/1/s/test/automation/out/task.js:68:13)
      at async Context.<anonymous> (out/areas/task/task-quick-pick.test.js:37:17)

macOS browser:

1) VSCode Smoke Tests (Web)
       Task
         Task Quick Pick
           Tasks: Run Task
             hide property - false:
     Error: Timeout: get elements '.quick-input-widget .quick-input-list .monaco-list-row .quick-input-list-row > .monaco-icon-label .label-name' after 20 seconds.
      at Code.poll (/Users/runner/work/1/s/test/automation/out/code.js:208:23)
      at async Code.waitForElements (/Users/runner/work/1/s/test/automation/out/code.js:161:16)
      at async QuickInput.waitForQuickInputElements (/Users/runner/work/1/s/test/automation/out/quickinput.js:28:9)
      at async Task.assertTasks (/Users/runner/work/1/s/test/automation/out/task.js:29:13)
      at async Context.<anonymous> (out/areas/task/task-quick-pick.test.js:34:17)

  2) VSCode Smoke Tests (Web)
       Task
         Task Quick Pick
           Tasks: Run Task
             icon - color only:
     Error: Timeout: get text content ' .monaco-editor[data-uri$="tasks.json"] .view-lines' after 20 seconds.
      at Code.poll (/Users/runner/work/1/s/test/automation/out/code.js:208:23)
      at async Code.waitForTextContent (/Users/runner/work/1/s/test/automation/out/code.js:152:16)
      at async Editor.waitForTypeInEditor (/Users/runner/work/1/s/test/automation/out/editor.js:79:9)
      at async Task.configureTask (/Users/runner/work/1/s/test/automation/out/task.js:68:13)
      at async Context.<anonymous> (out/areas/task/task-quick-pick.test.js:47:17)

  3) VSCode Smoke Tests (Web)
       Task
         Task Quick Pick
           Tasks: Run Task
             icon - icon & color:
     Error: Timeout: get text content ' .monaco-editor[data-uri$="tasks.json"] .view-lines' after 20 seconds.
      at Code.poll (/Users/runner/work/1/s/test/automation/out/code.js:208:23)
      at async Code.waitForTextContent (/Users/runner/work/1/s/test/automation/out/code.js:152:16)
      at async Editor.waitForTypeInEditor (/Users/runner/work/1/s/test/automation/out/editor.js:79:9)
      at async Task.configureTask (/Users/runner/work/1/s/test/automation/out/task.js:68:13)
      at async Context.<anonymous> (out/areas/task/task-quick-pick.test.js:52:17)

After examining these results, I have merged a PR which adds an optional callback to the dispatch method and also removes the 100ms timeout from the dispatchKeybinding method. For the smoke tests that fail below, I added a timeout of 100ms after the dispatchKeybinding call. The dispatchKeybinding has also been renamed to sendKeybinding to make it clearer that it is merely sending the keybinding.

As part of this change, it would be good if the owners of the smoke tests that send keybindings could review or add callbacks where necessary. It would be preferable to have callbacks everywhere so as to avoid smoke test flakiness.
For the failing smoke tests, it would be good if the owners could remove the 100ms timeout and add a callback which would ensure the keybinding is correctly processed.

Lines where to add a callback and remove the 100ms timeout

@meganrogge

[ ] test/automation/src/task.ts, line 37 to 39
[ ] test/automation/src/task.ts, line 63 to 65
[ ] test/automation/src/task.ts, line 86 to 88

@Tyriar

[ ] test/automation/src/terminal.ts, line 86 to 88
[ ] test/automation/src/terminal.ts, line 123 to 125
[ ] test/automation/src/terminal.ts, line 128 to 130
[ ] test/automation/src/terminal.ts, line 140 to 142

Lines where to review/add a callback

@connor4312

[x] test/automation/src/debug.ts, line 61 to 68
[x] test/automation/src/debug.ts, line 85 to 88
[x] test/automation/src/debug.ts, line 142 to 145

@jrieken

[ ] test/automation/src/editor.ts, line 39
[ ] test/automation/src/peek.ts, line 42

@bpasero

[ ] test/automation/src/editors.ts, line 15 to 17
[ ] test/automation/src/editors.ts, line 34
[ ] test/automation/src/editors.ts, line 65 to 70

@TylerLeonhardt

[ ] test/automation/src/quickinput.ts, line 37
[ ] test/automation/src/quickinput.ts, line 51 to 57

@lramos15

[ ] test/automation/src/explorer.ts, line 19 to 23

@ulugbekna

[ ] test/automation/src/keybindings.ts, line 15 to 20
[ ] test/automation/src/keybindings.ts, line 28 to 30

@roblourens

[ ] test/automation/src/notebook.ts, line 30
[ ] test/automation/src/notebook.ts, line 38

@sandy081

[ ] test/automation/src/quickaccess.ts, line 142 to 154
[ ] test/automation/src/quickaccess.ts, line 162

@lszomoru

[ ] test/automation/src/scm.ts, line 48

@osortega

[ ] test/automation/src/search.ts, line 43 to 48
[ ] test/automation/src/search.ts, line 71
[ ] test/automation/src/search.ts, line 75
[ ] test/automation/src/search.ts, line 80
[ ] test/smoke/src/areas/search/search.test.ts, line 26 to 32

@rzhao271

[ ] test/automation/src/settings.ts, line 27 to 28
[ ] test/automation/src/settings.ts, line 42 to 43
[ ] test/automation/src/settings.ts, line 51 to 53
[ ] test/automation/src/settings.ts, line 73 to 80
[ ] test/smoke/src/areas/preferences/preferences.test.ts, line 32
[ ] test/smoke/src/areas/preferences/preferences.test.ts, line 55 to 57

Please let me know if you feel the assignment is incorrect.

Mar 04 '25 08:03 aiday-mar

vscode vscode copied to clipboard

Review smoke test `sendKeybinding` usage

vscode
vscode copied to clipboard