[Feature]: Support running tests via the VSCode extension for supported browsers and the WebDriver BiDi protocol

[whimboo]

🚀 Feature Request

I would like to better debug existing Playwright tests for WebDriver BiDi and want to use the Playwright extension for VSCode. Sadly it doesn't work and seems to fail starting the actual browser.

Steps:

1. Clone Playwright and install the Playwright extension in VSCode
2. Go to the Test Explorer and add tests/bidi/playwright.config.ts as your config
3. Select all the browsers
4. Enable Show Browser
5. Go to one test and press Run Test

Each test is failing most likely because the browser cannot be started. The error that I get is:

TypeError: browserType.connect: Cannot read properties of undefined (reading 'launch')

Maybe this is because we need a specific version of the browser? If yes it would be good to at least allow to configure the path to the binary, so that it can be manually chosen.

Example

No response

Motivation

This would be helpful for me to easily test and debug issues with Playwright tests as seen with our developer builds of Firefox.

CC @yury-s, @mxschmitt

[yury-s]

https://github.com/microsoft/playwright/pull/36038 should make it work. I still see some exceptions from emulation.setGeolocationOverride when restarting tests from VS Code.

You may want to set something like that in the config to pick nightly binary and enable more logging as VSCode doesn't propagate some of the env correctly:

diff --git i/tests/bidi/playwright.config.ts w/tests/bidi/playwright.config.ts
index d650233e7..fc037421c 100644
--- i/tests/bidi/playwright.config.ts
+++ w/tests/bidi/playwright.config.ts
@@ -22,6 +22,10 @@ import { type Config, type PlaywrightTestOptions, type PlaywrightWorkerOptions,
 import * as path from 'path';
 import type { TestModeWorkerOptions } from '../config/testModeFixtures';
 
+process.env.PWDEBUGIMPL = '1';
+process.env.DEBUG = 'pw:browser';
+process.env.BIDI_FFPATH = '/Users/yurys/playwright/firefox/mac_arm-nightly_140.0a1/Firefox Nightly.app/Contents/MacOS/firefox';
+
 const headed = process.argv.includes('--headed');
 const trace = !!process.env.PWTEST_TRACE;
 const hasDebugOutput = process.env.DEBUG?.includes('pw:');

[whimboo]

@yury-s I assume that for your last comment I would still need the proposed PR 36038?

[yury-s]

Yes, that PR is still needed to make VS Code launch our Bidi tests as the Bidi browsers use different browser name than the bundled browsers. You can apply the change locally, it should work.

[whimboo]

Ok, so that works. Firefox can be launched and the test is now run. Thanks! Do you think there is a chance to get the other PR landed at some point?

[hbenl]

This is working for me when I add export BIDI_FFPATH="/Applications/Firefox Nightly.app/Contents/MacOS/firefox" to my ~/.zprofile and follow the steps in this issue's description.

[whimboo]

Thanks this works for me now as well. But one thing that I noticed is that when I want to debug Playwright core code it only works for the client but not the server. Therefore I have to use the global debugger feature and manually update a test with .only.

@yury-s would it be possible for the extension to support such a custom debugger configuration?

[hbenl]

@yury-s would it be possible for the extension to support such a custom debugger configuration?

We were thinking of something like this. In this configuration we could set autoAttachChildProcesses to true, which should allow us to debug the server code.

[yury-s]

cc @Skn0tt

[yury-s]

If you want to debug the server implementation, I'd suggest you do it by opening JavaScript Debug Terminal in VSCode (cmd+shift+p and type 'javascript debug terminal') and then starting the tests using a command line in the terminal (e.g. npm run ctest -- page-check:20). This way VSCode will auto attach to the node processes spawned by playwright and you will be able to stop on the breakpoints. Note that you need to set breakpoints in the compiled .js files rather than in the typescript sources, as v8 doesn't know about their source maps until the file is executed, i.e. set breakpoints in packages/playwright-core/lib/server/frames.js etc (once V8 loads the js file you may be able set breakpoints in the frames.ts file too).

The reason I don't recommend debugging the backend while starting the tests from the extension is that it changes the process interaction (and how the browser is launched) quite a bit and it will just complicate the debugging.

[Skn0tt]

Thanks for mentioning me - chiming in as i'm deep into our extension at the moment. I agree, with the architecture of how VS Code reuses the browsers + playwright server between tests, it's not feasible to attach a debugger to them. While that's cumbersome for working on Playwright itself, I don't see it as a problem for users.

[whimboo]

Oh, the JavaScript Debug Terminal is great! It allows debugging even without having to add specific configurations to launch.json. @yury-s please note that I can set breakpoints in TypeScript files and the debugger stops even when I did some code modifications. Most likely because I'm running npm run watch in the background? Or maybe something has changed in VSCode since you tried?

[whimboo]

Based on the feedback so far and given that we can successfully run tests and debug code as well in core Playwright we can most likely close this issue now?