test: add an end to end smoke test using vscode-extension-tester

[MilanKovacic]

1. Added end to end testing infrastructure using vscode-extension-tester
2. Added a basic smoke test

vscode-extension-tester will download the latest Visual Studio Code, package the extension, run it in isolated Visual Studio Code instance, and run the tests.

[sheremet-va]

Would be nice to also have it here: https://github.com/vitest-dev/vitest-ecosystem-ci (should just reference the name of the script in this repo)

To catch regressions in Vitest itself.

[hi-ogawa]

I haven't tried this but I found this one from webdriverio https://github.com/webdriverio-community/wdio-vscode-service, which looks similar/alternative to vscode-extension-tester than @vscode/test-cli.

I'm not familiar with neither selenium nor webdriverio, so I cannot tell the difference myself, but maybe it's worth comparing?

[sheremet-va]

I haven't tried this but I found this one from webdriverio webdriverio-community/wdio-vscode-service, which looks similar/alternative to vscode-extension-tester than @vscode/test-cli.

I'm not familiar with neither selenium nor webdriverio, so I cannot tell the difference myself, but maybe it's worth comparing?

I like their API! I wonder if Playwright has a similar thing 🤔 It's impossible to google because they have their own extension 😄

[MilanKovacic]

Both of them are very similar. The only difference is the underlying tech ("This project was highly inspired by the vscode-extension-tester project which is based on Selenium. This package takes the idea and adapts it to WebdriverIO."). vscode-extension-tester appears to be more popular.

It is possible to use Playwright, but then we would have to do the setup ourselves (downloading VSCode, launching it, getting access to the window, etc.), which is not really worth it in my opinion, as we will have just a smoke test.

We can use vscode-extension-tester as a starting point, and down the line check out the progress of other options (switching them is trivial).

[ffMathy]

Why use vs-code-extension-tester at all and not just rely on the official vscode-test which is already end-to-end and spins up a real vscode instance?

[MilanKovacic]

Why use vs-code-extension-tester at all and not just rely on the official vscode-test which is already end-to-end and spins up a real vscode instance?

vscode-test is not a true end to end testing solution. vscode-extension-tester, and wdio-vscode-service operate on a level of exposing the access to VSCode's page objects, which is suitable for a smoke test.

A lot of popular extensions use vscode-test primarily while using other tools for a smoke test, including:

https://marketplace.visualstudio.com/items?itemName=ritwickdey.LiveServer https://marketplace.visualstudio.com/items?itemName=nrwl.angular-console

...and even Microsoft's official Playwright extension: https://github.com/microsoft/playwright-vscode/blob/main/tests-integration/tests/basic.test.ts

[hi-ogawa]

Thanks for the links. Playwright one is very interesting.

• https://github.com/microsoft/playwright-vscode/blob/main/tests-integration/tests/baseTest.ts
• https://github.com/microsoft/playwright-vscode/pull/353
• https://github.com/microsoft/playwright/issues/22351

As you've already mentioned, essentially it's just "downloading VSCode, launching it, getting access to the window", but this is implemented as test fixture, so in principle, I think we could even do that as Vitest fixture too.

I think what vscode-extension-tester and wdio-vscode-service additionally do is to provide many builtin "proxy object" to easily access known UI components on Vscode view. It's getting interesting for me, so I'll probably dig into how these ideas are implemented later.

Personally, going with vscode-extension-tester is fine by me. I would wish there's a way to avoid additional tsc transpilation, but if you don't see a quick way, then that's fine by me as well https://github.com/vitest-dev/vscode/pull/270#discussion_r1503870167.

[hi-ogawa]

I went through playwright, wdio-vscode-service, and vscode-extension-tester and got a rough idea what they do, so let me share my little survey.

• playwright: they have experimental electron support where they provides library API to launch electron app and manipulate windows via their familiar Page interface. It's not integrated at CLI level, so there's no recording/debugging feature yet. This means you need to find an exact selector from complicated vscode ui by yourself, so it's practically impossible to write and maintain such tests.

• vscode-extension-tester: It manipulates vscode/electron app via selenium chrome webdriver, so they have Driver as a basic interface. It additionally provide a set of "page objects" utilities on top of Driver to easily manipulate known vscode UI elements. It also has cli to manage vscode/chrome binary and thin-wrapper to run mocha tests, but this package is actually usable as a library. As an example, I used this package and wrote e2e tests on Vitest https://github.com/hi-ogawa/vscode-extension-shell-shortcut/pull/12

• wdio-vscode-service: mostly same as vscode-extension-tester but built as webdriverio plugin (what they call launcher/worker service https://webdriver.io/docs/customservices). "page objects" utilities appear to be mostly same, so the difference of usability would probably come from the basic experience of webdriverio vs selenium. This package is tied to webdriverio integration, so I couldn't find a way to re-purpose this as a library unfortunately.

Though I went through all this, actually my quick take away from this is that, even with "page objects" utilities, it looks quite difficult to write tests in this way compared to vscode-test https://github.com/vitest-dev/vscode/pull/262 where test code runs directly in a extension process and you have a way more flexibility to manipulate vscode itself by import * as vscode from "vscode".

[MilanKovacic]

Hi, the aim was not to replace vscode-test, but to write a single end to end smoke test that truly tests the complete flow, for example:

• Extension can be installed via package
• Testing tab is visible when extension activates
• Test is visible in the testing tab
• Test can be ran

Maintenance of such test would be close to zero, while providing value through realistic flow.

[hi-ogawa]

Ah, don't worry. I totally understand the intention of this PR and it makes sense to have only bare minimal test for this. I went too long, but I thought it's just worth sharing and I didn't meant to change anything of this PR as is.

[hi-ogawa]

Sorry for yet another update. I was very wrong about playwright electron and it actually supports recorder/inspector already, so test authoring works quite nice. I updated my demo to replace vscode-extension-tester with playwright https://github.com/hi-ogawa/vscode-extension-shell-shortcut/pull/14. I still use Vitest as a test runner, so playwright is used as a library to spawn electron app.

I know it's not really relevant to this specific PR, but I hope it still has some value to survey the testing ecosystem as a Vitest team.

[sheremet-va]

We have tests running with Playwright now: https://github.com/vitest-dev/vscode/tree/main/test-e2e

Do you think this can be closed?

[MilanKovacic]

Closed.