Current limitations blocking Playwright's WebDriver BiDi adoption

[yury-s]

We've spent a couple weeks adding support for Bidi to Playwright and running Playwright tests with the new protocol. The good news is there has been a large progress in Bidi support in Firefox and Chromium, many Playwright tests are now passing. However, there is still a long list of issues that are blocking Playwright from adopting Bidi as the browser automation protocol. At this point the number of bugs we identified is too large to file individual reports for each of them, instead we'll use this issue to describe some of the major problems that we've discovered.

🎉 Current stats are the following:

• Chromium: 2368 / 3877 passed (3.2m)
• Firefox: 1469 / 3877 passed (4.1m)

Spec

• [ ] Provide a way to configure the viewport before a popup or new tab starts loading content. Currently, the browsingContext.SetViewport command can only be sent to an existing page, and there is no way to pause the load of a new page (e.g., opened via ctrl+click or a popup); these pages always load with the default viewport first.
• [ ] Allow to configure options(aka capabilities) such as proxy and acceptInsecureCerts per user context rather than per session, so that same browser process can be reused for testing with different options.
• [ ] Add support for isMobile emulation to enable fixed layout mode.
• [ ] Allow specifying a screen size that is different from the viewport size.
• [ ] Include support for landscape mode emulation.
• [ ] Add support for getting content quads to ensure accurate clicking inside transformed elements. This is particularly relevant for tests involving transformed elements, as seen in tests/page/page-click.spec.ts.
• [ ] Implement DOM.getFrameOwner and DOM.describeNode to enable converting between DOM node handles and Frame handles. This would improve the implementation of Locator.contentFrame() and FrameLocator.owner(), as the current workaround using contentWindow does not work with shadow DOM, as seen in the page/frame-frame-element.spec.ts tests.
• [ ] Introduce an event or method to track when a navigation is committed, beyond the current DOMContentLoaded and load events, which would be useful for some client requirements (e.g., 'should return when navigation is committed if commit is specified'). We need to know immediately if the navigation has failed or succeeded.
• [ ] Provide access to the request body (see issue: https://github.com/w3c/webdriver-bidi/issues/748).
• [ ] Provide access to the response body (see issue: https://github.com/w3c/webdriver-bidi/issues/747).
• [ ] Include request.resourceType support (see issue: https://github.com/w3c/webdriver-bidi/issues/704).
• [ ] Add suggestedFileName to browsingContext.downloadWillBegin.
• [ ] Enable retrieval of the body of a download.
• [ ] Add locale emulation (https://github.com/w3c/webdriver-bidi/issues/774), including:
  • accept-language headers
  • JS Date and Intl APIs
  • navigator.language
• [ ] Add support for timezone emulation (see issue: https://github.com/w3c/webdriver-bidi/issues/749).
• [ ] Improve user agent emulation to be per user context (see https://github.com/w3c/webdriver-bidi/issues/448#issuecomment-1944294296)
  • Interception is prohibitively expensive and therefore impractical.
  • Also not all requests are intercepted, e.g. WebSocket.
• [ ] Add support for setting extra HTTP headers per user context, same as user agent https://github.com/w3c/webdriver-bidi/issues/777
• [ ] Allow to call script.addPreloadScript per context, so that the script is always evaluated before navigation in the page. See e.g. 'init script should run only once in popup' test in page-add-init-script.spec.ts.
• [ ] Support offline mode emulation https://github.com/w3c/webdriver-bidi/issues/776
• [ ] Support emulation of disabled JavaScript https://github.com/w3c/webdriver-bidi/issues/746

These are just some of the issue that we discovered while looking at the first cut. There are many other issues that are currently blocked by the ones above.

Implementation

• [x] Firefox: browsingContext.setViewport fails intermittently with {"type":"error","id":14,"error":"unknown error","message":"AbortError: Actor 'MessageHandlerFrame' destroyed before query 'MessageHandlerFrameParent:sendCommand' was resolved","stacktrace":""}. See this bug report.
• [x] Firefox: page.setContent doesn't work because of 'Error: page.setContent: SecurityError: The operation is insecure.' in utility context. Bug report
• [ ] Firefox pointerDown/pointerUp/pointerDown/pointerUp in separate input.performActions commands don't generate dblclick event, see 'should dblclick the div' in page-mouse.spec.ts
• [ ] Firefox: script.realmDestroyed comes after browsingContext.contextDestroyed
• [ ] Firefox: script.evaluate exception have no details/message
• [ ] Firefox: browsingContext.contextDestroyed comes before input.performActions response, see 'should not throw UnhandledPromiseRejection when page closes'. I.e. some events can come after browsingContext was destroyed.
• [ ] Firefox: navigation id stays the same for different navigations, see 'should return from goto if new navigation is started'
• [x] Intermittent Error: Protocol error (script.evaluate): unknown error Message: AbortError: Actor 'MessageHandlerFrame' destroyed before query 'MessageHandlerFrameParent:sendCommand' was resolved - apparently fails if a command is sent before about:blank navigation finishes.
• [x] Firefox: lazy loading iframes are not reported (can be disabled in settings I think)
• [ ] Firefox: setViewport does not affect window.screen.width/height, matchMedia, see 'should emulate device width'
• [ ] Firefox: CSP tests are failing
• [x] Firefox: "url" not supported yet in network.continueRequest
• [ ] Firefox: no requestfailed event when it's cancelled by the page, see 'should not throw if request was cancelled by the page' in
• [ ] Firefox: 'should amend utf8 post data' - doesn't work with non-latin post data
• [ ] Firefox: 'redirected requests should report overridden headers' - does not allow to override headers on redirects
• [ ] Firefox: browsingContext.create sometimes hangs in parallel tests
• [ ] Firefox: browsingContext.downloadWillBegin is not fired
• [ ] Firefox: "Blocked request with id 15 not found" when sending network.continueWithAuth, authentication not working
• [x] Firefox: about:blank page is required for firefox to not close even in headless! (--silent didn't work)
• [x] Firefox: crash reporter dialog, can probably be fixed with settings.
• [ ] Firefox: need canonical settings for testing browser
• [x] Chromium: Method 'network.setCacheBehavior' is not implemented.

[whimboo]

To mention it here as well... Thanks a lot for all the details @yury-s! This is pretty helpful. As stated on the BiDi issue already I've created a Google document that contains all the above details, for collaborative editing. If you could find the time to at least roughly prioritize those features that would be great. You have a better idea what's most important, but I'll also dig into and see what could be the main issues with Firefox specifically to get more tests passing. Thanks!

[yury-s]

@whimboo I've assigned priorities to some items in the document. Since these are the first major issues we encountered during testing, and many are blockers for further progress, most are marked as P1 or P0. For clarity, I've added definition for the priority values used. Those are priorities from Playwright's perspective. For Browser implementation issues, I highlighted those that result in massive failures (and should be prioritized higher). Beyond that I'd defer prioritization of individual feature bugs to the browser maintainers.

[whimboo]

@yury-s some of the listed entries for Firefox are fixed in the meantime. We probably should get the list above updated (items checked) or fully rely on the Google document. It's hard to sync items on both sides - and here I don't have permissions to edit your comment for that purpose.

[yury-s]

@whimboo thanks for bringing this up, I went ahead and marked resolved items, they should know be in sync with the Google doc.

[whimboo]

@yury-s I tried to run some default tests by using the CDP protocol and I noticed that there is quite a difference in the number of tests between Chromium and Firefox. While for Chromium 11961 tests are run for Firefox these are only 3950. Could you please explain the difference? Also is it enough to only care about those 3950 tests which also get run for the bidi-test command?

[yury-s]

@yury-s I tried to run some default tests by using the CDP protocol and I noticed that there is quite a difference in the number of tests between Chromium and Firefox.

What command did you use to run the tests?

I see same number (4066) when running npm run biditest -- --project='bidi-firefox-nightly-*' and npm run biditest -- --project='bidi-chromium-*'. Note that these commands only execute core tests for playwright library under tests/library and tests/page. There is a lot of other tests for test runner, component testing, installation etc. Those are less relevant to the browser debugging protocol as the features they test are built on top of playwright "library" which provides core interface for browser automation. We don't run them as part of the bidi suite at the moment and your command probably ran some of those or included another configuration for the entire suite.

[whimboo]

Thank you Yury! This information was helpful. It's good to know that this subset of tests will be enough to run.

One thing I noticed which leads to differences in behavior especially for tests is that when I run BiDi tests the conditions for browserName == 'firefox' do not match because for these tests the browser name is actually _bidiFirefox. I assume that for tests that as well fail with firefox we can just ignore for bidi and mark it as such in the tests?

[yury-s]

One thing I noticed which leads to differences in behavior especially for tests is that when I run BiDi tests the conditions for browserName == 'firefox' do not match because for these tests the browser name is actually _bidiFirefox.

Yes, that's correct, we currently use different browser name to simplify implementation. AFAICT there are 139 tests that have special condition for browserName === 'firefox' (either mark as failed or custom expectation or skipped).

I assume that for tests that as well fail with firefox we can just ignore for bidi and mark it as such in the tests?

For some of them yes as they make no sense in Firefox, e.g. https://github.com/microsoft/playwright/blob/d029b03d9fc7d65329b47e08b9690b237bee30b7/tests/library/capabilities.spec.ts#L149-L165. But for others Bidi can probably do a better job than Playwright's current implementation, e.g. https://github.com/microsoft/playwright/blob/d029b03d9fc7d65329b47e08b9690b237bee30b7/tests/library/browsercontext-events.spec.ts#L60-L75 can work in Firefox as it does in other browsers.

[whimboo]

Yes, that makes sense. But the question would be if the browserName should really be prefixed with _bidi if a second flag like protocol could be used? I could update the expectations and provide a PR if that is known.

Also I can see that when running the bidi tests locally all the tests in library/chromium/ (and others) are run as well. I assume that those are really Chromium specific and most likely should be excluded from running when using WebDriver BiDi. If yes, I can as well update the testIgnore section for the bidi configuration.

[lutien]

Hey here, I wanted to let you know, that in Firefox we've landed the support for setting preload script per user contexts. Also, not sure if you need it, but maybe it could be useful, that we landed an option to subscribe to events per user contexts.

[lutien]

Hey again, I wanted to notify you, that we landed in Firefox an ability to set viewport configuration per user context.

[sadym-chromium]

FYI https://github.com/w3c/webdriver-bidi/issues/840 is addressed, making "Add suggestedFileName to browsingContext.downloadWillBegin." implementable.

[whimboo]

@yury-s maybe you could sync further items from your checklist with the Playwright document status in case you want to keep individual items listed in this GitHub issue? Thanks.

[lutien]

Hey again, FYI we landed a new platform API on browsingContext to emulate the geolocation: https://bugzilla.mozilla.org/show_bug.cgi?id=1951962, and a new WebDriver BiDi command to do this: https://bugzilla.mozilla.org/show_bug.cgi?id=1954992.

[sadym-chromium]

Allow to configure ... acceptInsecureCerts per user context ...

@yury-s are you sure acceptInsecureCerts per user context is a blocker? I don't see this option in the existing Playwright API, only a global testOptions.ignoreHTTPSErrors.

[whimboo]

@yury-s are you sure acceptInsecureCerts per user context is a blocker? I don't see this option in the existing Playwright API, only a global testOptions.ignoreHTTPSErrors.

It is an optional argument when creating a new user context: https://playwright.dev/docs/api/class-browser#browser-new-context-option-ignore-https-errors

[goloveychuk]

Seems like interception responses are available now https://github.com/w3c/webdriver-bidi/pull/877

[OrKoN]

I saw some tests fail with Response body is not available for requests in Bidi and waiting for events from workers. I think both are currently supported:

1. latest chromium-bidi has initial support for getting response body (same level as the current CDP)
2. for dedicated workers, there is no dedicated targets like in CDP, but worker realms should be reported. That allows evaluation and associating workers with frames, gettings console messages etc.

[OrKoN]

Also, I think the tests relying on the popup event (waiting for event "popup") should be implementable with WebDriver BiDi (originalOpener attribute would allow linking a new browsing context to the one that opened it).

[whimboo]

Also, I think the tests relying on the popup event (waiting for event "popup") should be implementable with WebDriver BiDi (originalOpener attribute would allow linking a new browsing context to the one that opened it).

@hbenl checked that and it's indeed the case. Holger, are there any tests that are still marked as timeout?

[whimboo]

With a lot of APIs already implemented for WebDriver BiDi in Firefox and Chrome, and having added required code to Playwright to support these, I wonder if we could make it easier for users of Playwright who want to use WebDriver BiDi as protocol to switch to it - instead of using Juggler. I filed https://github.com/microsoft/playwright/issues/37277 to get this discussed.

[OrKoN]

Also, I think the tests relying on the popup event (waiting for event "popup") should be implementable with WebDriver BiDi (originalOpener attribute would allow linking a new browsing context to the one that opened it).

This turned out to be a bug in chromium-bidi that should be fixed now. Workers can be supported in the following way https://github.com/microsoft/playwright/pull/37331 (although not sure if that is the preferred way or if I missed any edge cases).

[sadym-chromium]

• Chromium: 2368 / 3877 passed (3.2m)
• Firefox: 1469 / 3877 passed (4.1m)

@yury-s how can we get the up-to-date data on this metrics?

[whimboo]

The current CI jobs for BiDi, which run daily, can be found at: https://github.com/microsoft/playwright/actions/workflows/tests_bidi.yml

As per the latest results we have:

• Chrome: 3098 / 4113 passed (38 min)
• Firefox: 3422 / 3995 passed (45 min)

[hbenl]

@sadym-chromium FYI: We've created a dashboard for the Playwright BiDI tests. You can filter out test files that you're not interested in by using the gear icon in the top left and bookmark the filtered view. Clicking on one of the bars in the chart takes you to a view for that test run and clicking on the "Diff" link in that view shows the changed test results compared to the previous test run.

[sadym-chromium]

FYI the network offline emulation is specified and implemented in Chromium

[dhruv-dabhi-st7]

It would be very helpful if Playwright provided an official WebDriver BiDi support matrix, similar to Puppeteer’s list here: Puppeteer Webdriver. This would clearly show which Playwright features are supported, partially supported, or missing when using BiDi (network interception, request/response bodies, viewport/emulation, downloads, DOM/shadow DOM, etc.). Right now the only reference is this issue, so having a dedicated matrix would greatly improve clarity and planning for users adopting BiDi even it's in experimental.