microsoft
[Feature]: Allow to configure the protocol (`juggler` or `webDriverBiDi`) when launching the browser
🚀 Feature Request
Now that WebDriver BiDi implements many of the requirements needed by Playwright, it would be great to see early adoption from users. Real-world feedback would be invaluable in helping us prioritize future API additions and enhancements.
I recently attempted to migrate another project from using Firefox with Juggler to WebDriver BiDi. However, it failed when trying to import _bidiFirefox from the playwright module. Besides the import issue itself, the use of the underscore prefix (_bidiFirefox) is also problematic. It typically indicates an internal or non-public API, which makes its usage feel unsupported or fragile.
Would it be possible to introduce a more stable and ergonomic way to choose between Juggler and WebDriver BiDi, perhaps via the launch options? For example, a protocol option could be added, allowing values like juggler (default) or webdriverBiDi. Internally, this would determine whether the firefox or _bidiFirefox browser class is instantiated. But maybe there is another option preferred by you?
Example
No response
Motivation
This type of setup worked well for us during the experimental phase with Puppeteer and would likely improve usability and adoption here as well.
@yury-s what do you think?
Discussed this in the team meeting today. Before we expose Bidi as a public API, we’d like a clearer picture of where support currently stands.
When we were adding WebKit and Firefox support, we relied on a dashboard showing the number of passing tests per browser, plus a breakdown of failing test files and their pass/fail counts. That visibility helped identify which areas were still unsupported.
For Bidi, the best signal we have right now is the raw test output (e.g. 3176 passed out of 4182). While that points to 1k+ failures, it doesn’t show which components are affected. I know you’ve tracked this in docs/spreadsheets before, so you probably already have good insights.
Could you set up a dashboard for Bidi Firefox that shows current
- passing / failed / skipped (e.g. due to timeout) / total
- breakdown of failures by test file
That way we can see what’s missing and if it’s ready to go public.
Discussed this in the team meeting today. Before we expose Bidi as a public API, we’d like a clearer picture of where support currently stands.
@yury-s, thank you for the feedback! So lets get on par to see at which state BiDi can be considered a public protocol for Playwright. Do you have specific guidelines / rules for that, like say are you differentiating between officially supported protocols and experimental ones? Or is it all the same for you?
When we were adding WebKit and Firefox support, we relied on a dashboard showing the number of passing tests per browser, plus a breakdown of failing test files and their pass/fail counts. That visibility helped identify which areas were still unsupported.
Could you set up a dashboard for Bidi Firefox that shows current
* passing / failed / skipped (e.g. due to timeout) / total * breakdown of failures by test file
This sounds like a good idea. Right now we only track the numbers in the Playwright via WebDriver BiDi Google document and spreadsheet, but having an automated way to update the current status would be kinda great. When you say that you were using a dashboard in the past, is that still accessible and could be adapted for WebDriver BiDi? It would be faster to setup compared to writing a new one from scratch.
For Bidi, the best signal we have right now is the raw test output (e.g. 3176 passed out of 4182). While that points to 1k+ failures
Hm, where do you have those numbers from? From the daily CI runs we currently have:
2025-09-15: 3246 / 3852 passed (42min) - 221 skipped
This means only about 600 tests are not passing at the moment, which is 60% compared to your provided numbers. Or did you consider to include more tests beside those we originally set and tracked until now?
, it doesn’t show which components are affected. I know you’ve tracked this in docs/spreadsheets before, so you probably already have good insights.
I've spent some time to collect the information as best as I could and added the API: WebDriver BiDi sheet to the Google spreadsheet. Could you please take a look and check the following items?
- Do I miss some important API or are some of them as listed not required (I marked some with question marks where I'm not sure - maybe you can add a comment with an explanation and tick of the checkbox of the
Playwright wantedcolumn?) - Are the priorities set correctly? Feel free to adjust as you think it is important. We are going to plan our next milestone soon so it would be good to know where our focus should be next.
Thanks!
@yury-s, thank you for the feedback! So lets get on par to see at which state BiDi can be considered a public protocol for Playwright. Do you have specific guidelines / rules for that, like say are you differentiating between officially supported protocols and experimental ones? Or is it all the same for you?
Playwright doesn’t distinguish between different “protocols” for driving the same browser today, those are internal details. We’d rather not put that choice on users.
We also don’t have strict rules for when something is ready for public. In the past, when adding support for Firefox and WebKit, we used dashboards comparing feature parity across browsers, and the decision came down to common sense. The API surface has grown since then, so many smaller features aren’t as critical. A dashboard that groups failing tests by component (per-file granularity should be fine) is still the best way to judge maturity. Core interactions (page navigation, selectors, mouse/keyboard input) are essential, while some of the niche APIs (CSP bypass, geolocation emulation, etc.) could be marked “not yet supported.”
What we can do is to introduce (initially hidden as we do today, then experimental, eventually public) new channels for Firefox, similar to the ones for Chrome/Edge. The channels are essentially existing 'moz-firefox', 'moz-firefox-beta' and 'moz-firefox-nightly'. The users can opt into the them and use Bidi to connect to the official Firefox builds. Since most of the Bidi code lives in the Mozilla repo, and we can’t patch it as quickly as Playwright-owned code (getting consensus from the Firefox engineers and the code reviews, in more complex cases passing it through the spec, etc.), it makes sense for the Firefox team to own those channels. This would avoid Playwright being blocked by other teams'/committees reaction time, staffing, or approval processes when quick turnaround is needed, and ensures Bidi has a single clear owner.
By default we’d continue shipping Juggler with our bundled Firefox build to maintain the current support guarantees. Over time, when Firefox Bidi implementation matures, the adoption grows and we can be confident that incoming bugs affecting Firefox Bidi are fixed in a timely fashion, we can revisit switching the default implementation from Juggler to Bidi.
How does that sound to you?
When you say that you were using a dashboard in the past, is that still accessible and could be adapted for WebDriver BiDi? It would be faster to setup compared to writing a new one from scratch.
The source code lives here https://github.com/aslushnikov/isplaywrightready. It had been written before we implemented our own test runner with its reporting APIs. It should be much easier to produce the similar data with Playwright today. I'd not look at the old code much and just vibe code a new dashboard from scratch.
This means only about 600 tests are not passing at the moment, which is 60% compared to your provided numbers. Or did you consider to include more tests beside those we originally set and tracked until now?
You are right, I looked at the total number of our Firefox tests with Juggler. You numbers for the BiDi (moz-firefox-nightly) bot are more accurate, what I see is:
3852 total
385 failed
221 skipped
3246 passed
I remember that some of the Firefox tests, that have been excluded from the original suite for Bidi, cover Playwright functionality that is not browser specific. Shouldn't they be mostly passing now that more features have been supported? It's important for the users that the trace viewer/inspector work with Firefox Bidi. If the trace-viewer tests are failing because of non-implemented pieces of the API, it's a strong signal that the API is of high priority.
I've spent some time to collect the information as best as I could and added the API: WebDriver BiDi sheet to the Google spreadsheet. Could you please take a look and check the following items?
From what I see, I believe "Actions: Scroll to element" corresponds to scrollIntoViewIfNeeded and is of top priority, as it used in all actions. The rest look okay, having a dashboard with the failing tests would help to identify what's missing and refine the list.
We also don’t have strict rules for when something is ready for public. In the past, when adding support for Firefox and WebKit, we used dashboards comparing feature parity across browsers, and the decision came down to common sense. The API surface has grown since then, so many smaller features aren’t as critical. A dashboard that groups failing tests by component (per-file granularity should be fine) is still the best way to judge maturity. Core interactions (page navigation, selectors, mouse/keyboard input) are essential, while some of the niche APIs (CSP bypass, geolocation emulation, etc.) could be marked “not yet supported.”
We are going to explore a solution for a dashboard so that it's easier to examine which APIs, that are required, are still missing or haven't been fully implemented yet. We will let you know when there is a dashboard available.
What we can do is to introduce (initially hidden as we do today, then experimental, eventually public) new channels for Firefox, similar to the ones for Chrome/Edge. The channels are essentially existing 'moz-firefox', 'moz-firefox-beta' and 'moz-firefox-nightly'.
That seems fine. Based on the fact that those channels are hidden as of today, is there a way for clients using Playwright as framework to make use of them? I tried that but it fails with Unsupported firefox channel "moz-firefox". I assume that's expected. When I'm trying to use _bidiFirefox it fails because there is no such exported name. Would it work when the support of the channels is marked as experimental? Which expectations do you then have for experimental, and public? For us it would be good to get some feedback from actual users so that we can do our planning more based on actual users needs as just on the assumption of unit tests.
You are right, I looked at the total number of our Firefox tests with Juggler. You numbers for the
BiDi (moz-firefox-nightly)bot are more accurate, what I see is:3852 total 385 failed 221 skipped 3246 passed
Are you actually uploading results for Firefox with Juggler as artifacts as well on a daily basis? Or is that something you aren't doing right now? If might be good to as well compare the results from Juggler with BiDi.
I remember that some of the Firefox tests, that have been excluded from the original suite for Bidi, cover Playwright functionality that is not browser specific. Shouldn't they be mostly passing now that more features have been supported? It's important for the users that the trace viewer/inspector work with Firefox Bidi. If the trace-viewer tests are failing because of non-implemented pieces of the API, it's a strong signal that the API is of high priority.
Yes, most of these tests are passing now, and we got them enabled in the meantime. Some of them are still failing but we have them on our list now.
I've spent some time to collect the information as best as I could and added the API: WebDriver BiDi sheet to the Google spreadsheet. Could you please take a look and check the following items?
From what I see, I believe "Actions: Scroll to element" corresponds to scrollIntoViewIfNeeded and is of top priority, as it used in all actions. The rest look okay, having a dashboard with the failing tests would help to identify what's missing and refine the list.
For Scroll to Element there are two different scenarios. One you already mentioned for Actions, so that a given action in the chain can scroll to the element first, before interacting with it. The other scenario is a global Scroll into View feature, which for the time being could be implemented with a script evaluation call if needed. Which of those two are you referring to with a high priority?
That seems fine. Based on the fact that those channels are hidden as of today, is there a way for clients using Playwright as framework to make use of them? I tried that but it fails with Unsupported firefox channel "moz-firefox". I assume that's expected. When I'm trying to use _bidiFirefox it fails because there is no such exported name. Would it work when the support of the channels is marked as experimental? Which expectations do you then have for experimental, and public? For us it would be good to get some feedback from actual users so that we can do our planning more based on actual users needs as just on the assumption of unit tests.
I think we should be able to update the code to make playwright.firefox work with the new Firefox channels, similar to how it works for Chrome and MSEdge on playwright.chromium. Ideally we shouldn't need separate playwright._bidiFirefox (it was more convenient to have it separate for the initial implementation), even if under the hood it uses 2 different implementations.
Are you actually uploading results for Firefox with Juggler as artifacts as well on a daily basis? Or is that something you aren't doing right now? If might be good to as well compare the results from Juggler with BiDi.
We do upload test results to our internal flakiness dashboard for tracking general health of the project. It does provide way more detailed view than what is needed for comparing feature parity of Juggler v Bidi though. You can add a step that would upload juggler results to another place for comparison.
For Scroll to Element there are two different scenarios. One you already mentioned for Actions, so that a given action in the chain can scroll to the element first, before interacting with it. The other scenario is a global Scroll into View feature, which for the time being could be implemented with a script evaluation call if needed. Which of those two are you referring to with a high priority?
I'm talking about scrolling the element before an action, as this is what we do before each mouse interaction. I'm not sure how global Scroll into View is different, but if you say it could be scripted then it's likely lower priority.
That seems fine. Based on the fact that those channels are hidden as of today, is there a way for clients using Playwright as framework to make use of them? I tried that but it fails with Unsupported firefox channel "moz-firefox". I assume that's expected. When I'm trying to use _bidiFirefox it fails because there is no such exported name. Would it work when the support of the channels is marked as experimental? Which expectations do you then have for experimental, and public? For us it would be good to get some feedback from actual users so that we can do our planning more based on actual users needs as just on the assumption of unit tests.
I think we should be able to update the code to make
playwright.firefoxwork with the new Firefox channels, similar to how it works for Chrome and MSEdge onplaywright.chromium. Ideally we shouldn't need separateplaywright._bidiFirefox(it was more convenient to have it separate for the initial implementation), even if under the hood it uses 2 different implementations.
That would be great! Do you have a rough idea of when this could be done? Also, I’m still a bit unsure about whether this depends on having the full dashboard in place and the requested test coverage for certain not-yet-fully-implemented APIs. Is that a hard requirement even for offering experimental support? Or would partial coverage be acceptable in that case? We would really be interested to get user reports if possible.
Are you actually uploading results for Firefox with Juggler as artifacts as well on a daily basis? Or is that something you aren't doing right now? If might be good to as well compare the results from Juggler with BiDi.
We do upload test results to our internal flakiness dashboard for tracking general health of the project. It does provide way more detailed view than what is needed for comparing feature parity of Juggler v Bidi though. You can add a step that would upload juggler results to another place for comparison.
Thanks! We will discuss internally if we want to use it.
For Scroll to Element there are two different scenarios. One you already mentioned for Actions, so that a given action in the chain can scroll to the element first, before interacting with it. The other scenario is a global Scroll into View feature, which for the time being could be implemented with a script evaluation call if needed. Which of those two are you referring to with a high priority?
I'm talking about scrolling the element before an action, as this is what we do before each mouse interaction. I'm not sure how
global Scroll into Viewis different, but if you say it could be scripted then it's likely lower priority.
One possible path forward would be to implement this directly in Playwright using plain script evaluation, rather than relying on a dedicated BiDi API. So the previously mentioned Playwright API would still be exposed to clients, just with a different underlying mechanism. If that approach is acceptable, we could look into supporting it for both Firefox and Chrome.
That would be great! Do you have a rough idea of when this could be done? Also, I’m still a bit unsure about whether this depends on having the full dashboard in place and the requested test coverage for certain not-yet-fully-implemented APIs. Is that a hard requirement even for offering experimental support? Or would partial coverage be acceptable in that case? We would really be interested to get user reports if possible.
I will work on it, don't remember what the blockers were that made us use _bidiFirefox instead. We don't need to block this on the dashboard and the readiness criteria. We can add the new channel without advertising it in the docs for now, it should allow to collect the user feedback sooner. For the official channel (even if experimental) with the docs, it'd be nice to have the dashboard to be transparent on what is ready and what is not.
One possible path forward would be to implement this directly in Playwright using plain script evaluation, rather than relying on a dedicated BiDi API. So the previously mentioned Playwright API would still be exposed to clients, just with a different underlying mechanism. If that approach is acceptable, we could look into supporting it for both Firefox and Chrome.
You can try and see if that works, maybe you can leverage some of the Firefox APIs to avoid the native part altogether. There were some difficulties around minimal offsets and display:contents, probably something else, but we should have tests for that. If the user land implementation doesn't work for some of the edge cases, you can still start with that. I'd not bother with reimplementing Chrome as CDP supports the scrolling with a native command.
That would be great! Do you have a rough idea of when this could be done?
After https://github.com/microsoft/playwright/pull/37696 the same playwright.firefox instance can be used to connect to Firefox over bidi. Please give it a try, there might be some regressions.
@yury-s FYI: We've created a dashboard for the Playwright BiDI tests. You can filter out test files that you're not interested in by using the gear icon in the top left and bookmark the filtered view. Clicking on one of the bars in the chart takes you to a view for that test run and clicking on the "Diff" link in that view shows the changed test results compared to the previous test run.
Nice! A few thoughts:
- there are some failing click tests, we'd consider them a top priority
- other interaction tests like mouse, drag, keyboard also cover user interactions and would be the top priority for fixing before opening this to a broader audience. Most of the test cases are based on the client reports, so we know that they are actually important for real world scenarios even if they look like unlikely edge cases.
- network events and interception are the next big things there, it'd be nice to have them fixed
- it'd be easier to assess failures in each area/file if
- each file had a number of failing/total tests and
- could be expanded to see which tests are failing
- It doesn't make much sense to have Firefox and Chrome on the same dashboard. If we want to compare, it should be Firefox Bidi vs Firefox juggler.
* it'd be easier to assess failures in each area/file if * each file had a number of failing/total tests and
they're shown in the tooltip when hovering over the results bar
* could be expanded to see which tests are failing
I've just added this
That would be great! Do you have a rough idea of when this could be done?
After #37696 the same
playwright.firefoxinstance can be used to connect to Firefox over bidi. Please give it a try, there might be some regressions.
Thanks a lot @yury-s. It looks good with an example script within this repository. But when I have a dependency on Playwright in another project and install 1.56.0 I cannot get it to work. I can see the following difference:
In this repository we have (link):
const registryExecutable = registry.findExecutable(this.getExecutableName(options));
if (!registryExecutable || registryExecutable.browserName !== this._name)
But when having the Playwright 1.56.0 package installed:
const registryExecutable = import_registry.registry.findExecutable(this.getExecutableName(options));
if (!registryExecutable || registryExecutable.browserName !== this._name)
throw new Error(`Unsupported ${this._name} channel "${options.channel}"`);
Do you know where this difference is coming from? As result I get Unsupported firefox channel "moz-firefox.
But when I have a dependency on Playwright in another project and install 1.56.0 I cannot get it to work.
The change didn't make it to 1.56 (these lines were removed in the PR but still present in the 1.56 branch). It will be released in 1.57. Before that you can give it a try with the canary (@next) builds.
Do you know where this difference is coming from? As result I get Unsupported firefox channel "moz-firefox.
This the result of transformation from typescript to a cjs javascript module. This is working as expected. See utils/build/build.js for more details.
The change didn't make it to 1.56 (these lines were removed in the PR but still present in the 1.56 branch). It will be released in 1.57. Before that you can give it a try with the canary (@next) builds.
Oh, I see - it explains it. Let me then wait until the 1.57.0 release before trying it again with the other project. Thanks for letting me know!
We reviewed several Firefox Bidi test failures (top pages of the dashboard) and tried to break it down into the following categories:
P1 - Core functionality. Required for a complete product; we wouldn’t ship without it. P2 - Important edge-case functionality; not strictly blocking, but absence will trigger noticeable user complaints. P3 - Niche functionality; affects only a small subset of users.
-
P1
- csp - important to users
- network-post-data - providing a separate command for getting post data as a stream would be ideal
- page-filechooser
- page-download
- popup
- page-event-console
- page-request-continue - We are going to address https://github.com/microsoft/playwright/issues/38108 and update our expected behavior in Playwright. After that fixing this test for Bidi should be straightforward.
- page-set-extra-http-headers - needs to be low level, zero overhead. Having it implemented based on the network interception is impractical due to the perf overhead.
- browser-context-set-extra-http-headers - same as above
- video - migrate everything to screencast
-
P2
- elementhandle-scroll-into-view
- geolocation
- page-drag
- page-emulate-media
- page-request-gc - upstream playwright implementation?
- permissions
- web-socket - inspection
- workers
-
P3
- browsercontext-cookies-thirdparty
- jshandle-to-string - investigate, [suspicious]
- page-event-crash - no crash events is ok
- pdf - chromium specific in Playwright, but can be implemented with Print command
- tap
- locator-dispatchevent-touch - implemented in injected script, should work in BiDi. On a closer look,
TouchandTouchEventare not in the global context in Firefox because touch emulation is not supported apparently. This goes into P3 bucket because we don't support mobile emulation for Firefox and touch is most important on mobile devices.
We reviewed several Firefox Bidi test failures (top pages of the dashboard) and tried to break it down into the following categories:
P1 - Core functionality. Required for a complete product; we wouldn’t ship without it. P2 - Important edge-case functionality; not strictly blocking, but absence will trigger noticeable user complaints. P3 - Niche functionality; affects only a small subset of users.
* **P1** * csp - important to users
@yury-s I am trying to work on CSP related issues at the moment, and I would need more details about what exactly is P1 around CSPs.
At the moment since Playwright uses eval to wrap any script passed to Playwright's page.evaluate, any call to page.evaluate on a page which uses CSPs preventing eval/new Function will fail. IMO that's the most important issue around CSPs at the moment because it's the one which is the easiest for users to encounter.
There is a second scenario: if the script provided by the user also contains eval or new Function. Since PW page.evaluate also uses eval this ends up nesting two evals which is a slightly more problematic use case to handle on our side.
And finally there is a bypassCSP argument which only seems to be used for addScriptTag. So far, my patches to address the second scenario of nested evals also happen to bypassCSPs for addScriptTag, so it might be difficult to discriminate only against addScriptTag for us. And from a spec perspective I imagine it will be hard to have a configuration to bypass CSPs which only applies to a small subset of CSP-impacted features. So this might require more discussions upstream.
What do you consider P1 between:
- usage of page.evaluate on any page with a CSP
- usage of page.evaluate with eval/new Function on any page with a CSP
- support for the bypassCSP option
Thanks
All of the above are important. bypassCSP: true should make the user code (inline scripts in the page and also "nested evals") work without CSP violations, e.g. adding a script tag with a wrong nonce should not throw and should work as if there was no CSP on the page.
Our tests about bypassCSP option are bad as they only check that addScriptTag works. That method (as well as page.evaluate etc) should work regardless of bypassCSP flag. I'm updating the tests in https://github.com/microsoft/playwright/pull/38295, this should clarify the expectations of the option's behavior.
Ok thanks for the feedback.
@yury-s Can you share more insights about the added value of the bypassCSP option, if everything coming from the protocol script evaluation already bypasses CSPs? The idea is that the website would be hosting scripts / features which are not valid for their own CSPs but want to trigger them during automation? That seems very niche, and I don't really when / how this would be useful. Am I missing something? Since we'll need to discuss it on the spec side first, I would prefer to understand the added value before I bring it to the group.
@yury-s Second question: at the moment, Playwright is bypassing CSPs by default for page.evaluate, but this only works for synchronous code. For instance a test such as
it.only('should work with async new Function() and CSP', async ({ page, server }) => {
server.setCSP('/empty.html', 'script-src ' + server.PREFIX);
await page.goto(server.PREFIX + '/empty.html');
expect(await page.evaluate(
async () => {
await new Promise(r => setTimeout(r, 1000));
return new Function('return true')();
}
)).toBe(true);
});
fails on Chrome / Firefox with playwright. This is a limitation from the current CDP implementation I already highlighted in https://github.com/w3c/webdriver-bidi/issues/1024 . If we are supposed to mimic the same behavior on Firefox, we need to know if disabling CSPs only synchronously is intentional or something you want to change in the future. IMO this makes for a very inconsistent feature, because that means eval/new Function only work in some cases during script evaluation, but it does make the implementation on the browser side easier if we keep it synchronous. Do we agree that the CSP bypass should only apply to synchronous code spawned from script.evaluation or is this something you aim to change in the future.
@yury-s Can you share more insights about the added value of the
bypassCSPoption, if everything coming from the protocol script evaluation already bypasses CSPs? The idea is that the website would be hosting scripts / features which are not valid for their own CSPs but want to trigger them during automation? That seems very niche, and I don't really when / how this would be useful. Am I missing something? Since we'll need to discuss it on the spec side first, I would prefer to understand the added value before I bring it to the group.
An example that comes to my mind: during local development you update inline script, but don't want to bother updating it's sha nonce while still iterating. If the CSP is enabled the app will start failing. Same applies to adding new script tags in the app. bypassCSP reduces the friction during the development phase by removing the security errors out of the way. When the change is ready, proper CSP will be configured. You can also dig up the history of the feature and see the user requests related to it.
@yury-s Second question: at the moment, Playwright is bypassing CSPs by default for
page.evaluate, but this only works for synchronous code. For instance a test such asit.only('should work with async new Function() and CSP', async ({ page, server }) => { server.setCSP('/empty.html', 'script-src ' + server.PREFIX); await page.goto(server.PREFIX + '/empty.html'); expect(await page.evaluate( async () => { await new Promise(r => setTimeout(r, 1000)); return new Function('return true')(); } )).toBe(true); }); fails on Chrome / Firefox with playwright. This is a limitation from the current CDP implementation I already highlighted in w3c/webdriver-bidi#1024 . If we are supposed to mimic the same behavior on Firefox, we need to know if disabling CSPs only synchronously is intentional or something you want to change in the future. IMO this makes for a very inconsistent feature, because that means eval/new Function only work in some cases during script evaluation, but it does make the implementation on the browser side easier if we keep it synchronous. Do we agree that the CSP bypass should only apply to synchronous code spawned from script.evaluation or is this something you aim to change in the future.
This is likely just an oversight or was likely done this way because bypassCSP for async operations would be much more complicated. In any case, skimming through the issues related to CSP I don't see any requests for support of async calls. Playwright API is largely driven by user feedback and since there haven't been any user complaints about lack of async operations support, we are not planning to change this. From the spec perspective, you probably want to be more proactive and cover the async case as well, since it is theoretically possible.