Support for disabling scripts from WebDriver BiDi

[juliandescottes]

What is the issue with the HTML Standard?

Attempt to summarize requirements for disabling scripts from WebDriver BiDi (see https://github.com/whatwg/html/pull/11441)

BiDi exposes the command emulation.setScriptingEnabled which allows to disable script execution either for a set of navigables or a set of user contexts. The goal is to allow to test the behavior of a page when visited by a user with JavaScript disabled, so it should replicate as faithfully as possible how the browser behaves in this case.

However WebDriver BiDi also exposes several commands to execute javascript in the context of a given page:

• script.evaluate and script.callFunction will execute a specific expression (or function) when the command is called
• script.addPreloadScript will register a script for a given set of navigables or user contexts, which will run before any other script when a page loads

Those commands should still be allowed to successfully execute JavaScript even against a page where JavaScript was disabled via emulation.setScriptingEnabled. It is important to allow test and tool authors to run JavaScript, because it is a very useful escape hatch to either fetch information from the page or interact with the page, in case there is no specific webdriver bidi command to cover what the author wants to do.

Since those additional scripts still run in the context of the page, they normally would rely on the event loop / finalization registry of the page. In order for those scripts to behave correctly, users would expect thenables to work and weakmaps to be properly cleanedup, but that is problematic since those would effectively not work for scripts from the content page. See questions in the corresponding PR eg https://github.com/whatwg/html/pull/11441#issuecomment-3213299602

cc @zcorpan @sadym-chromium

Let me know if I need to expand and add more details on a specific topic.

[zcorpan]

Thanks for filing this.

As I understand it, the use case is to be able to execute scripts from WebDriver BiDi (after disabling scripting), including following async operations. The fact that the page's scripts also are able to run such scripts is not needed for the use case, just a side-effect.

Questions:

• Is it OK to let such scripts run?
• Is the expectation that WebDriver BiDi tests disable scripts and then reload the page? In this case, the reloaded page is not able to create promises or weakmaps.
• Otherwise, is it possible to differentiate between thenables and weakmap finalizations created by the page and those created by WebDriver BiDi, and prevent the former from running?

[juliandescottes]

I would like to see feedback from @sadym-chromium but on my side:

Is it OK to let such scripts run?

From a user perspective, I think it's fine. If a WebDriver BiDi evaluated script happens to trigger a script / function created by the page earlier, it's ok.

Is the expectation that WebDriver BiDi tests disable scripts and then reload the page? In this case, the reloaded page is not able to create promises or weakmaps.

With the webdriver BiDi command, you can disable JS and then run tests with JS disabled without reloading the page. But reloading the page is still possible and JS should still be disabled after the reload. We keep track of this either per user context or per navigable, so it should be persistent across reloads. So if on the HTML spec side it implies that promises/weakmaps can't be used for any script evaluated in this page, we need an escape hatch, because we can't cripple script evaluation after a reload, that would be completely unexpected for users of this API. Is it not possible to still allow creating promises / weakmaps in case script was disabled by WebDriver BiDi?

Otherwise, is it possible to differentiate between thenables and weakmap finalizations created by the page and those created by WebDriver BiDi, and prevent the former from running?

Are you asking this from a technical perspective? For thenables, I feel like this should be possible, in the same wayd as we keep track of user activation?

[zcorpan]

From a user perspective, I think it's fine. If a WebDriver BiDi evaluated script happens to trigger a script / function created by the page earlier, it's ok.

OK, then I suppose it's fine to specify it, but we should have notes to call it out.

With the webdriver BiDi command, you can disable JS and then run tests with JS disabled without reloading the page. But reloading the page is still possible and JS should still be disabled after the reload. We keep track of this either per user context or per navigable, so it should be persistent across reloads. So if on the HTML spec side it implies that promises/weakmaps can't be used for any script evaluated in this page, we need an escape hatch, because we can't cripple script evaluation after a reload, that would be completely unexpected for users of this API. Is it not possible to still allow creating promises / weakmaps in case script was disabled by WebDriver BiDi?

I meant that it would be OK to run all thenables and weakmap finalizations in this context, because the page's scripts didn't run and so didn't have an opportunity to create promises or weakmaps.

It sounds like there are 3 scenarios that should be supported:

• First disable scripting, then load the page.
• Load the page, then disable scripting, then reload the page.
• Load the page, then disable scripting.

The third scenario means it's relevant what happens to page-created thenables and weakmap finalizations.

Are you asking this from a technical perspective? For thenables, I feel like this should be possible, in the same wayd as we keep track of user activation?

Yes, but also if it's acceptable in terms of implementation complexity, perf, and memory overhead.

From a theoretical purity standpoint, it would make sense to not run page-created scripts of any kind after WebDriver BiDi has disabled scripting. But it's possible that doing so is not reasonable in practice, and implementation concerns trump theoretical purity.

[juliandescottes]

We discussed this topic yesterday during the webdriver WG meeting. Trying to summarize, on the WebDriver side, we are fine with the fact that some user scripts (linked to thenables or finalization for instance) might still run after scripting was disabled via WebDriver BiDi.

Implementation wise, on Chrome side it seems difficult/impossible to check if a given script running via a thenable / finalization comes from a source which should be disabled (content page) or not (bidi script evaluation). On Firefox side it might be doable but challenging.

@zcorpan does that clarify the remaining questions?

@sadym-chromium @jgraham anything I forgot?

[sadym-chromium]

In chromium, we cannot check the origin of a microtask scheduled by setTimeout, setInterval or a finalizer. From the other side, we can check it for thenables.

In practice in chrome, if the JS is disabled, we don't run setInterval nor setTimeout from any source, including WebDriver/CDP scripts. We do run Finalizer regardless of the source of the script (CDP, WebDriver, page script), which allows for garbage collection.

[sadym-chromium]

Here is breakdown of the behaviors in case of scenario "Load the page, then disable scripting".

Macrotasks

Timer initialization steps (setTimeout, setInterval)

Current Spec (RUN)

• Timer initialization steps creates task.
• The task for a given handler:
  • RUN. (step 9.5) Invoke the handler.
  • (step 9.6.8) Create a classic script with handler.
    • DO NOT RUN. Check if scripting is disabled and sets source to the empty string if so.

Proposed Spec (DO NOT RUN)

• Timer initialization steps creates task.
• The task for a given handler:
  • DO NOT RUN. (step 9.5) Check if scripting is disabled and abort execution is so.

Chromium implementation (DO NOT RUN)

FinalizationRegistryCleanupJob

Current Spec (DO NOT RUN)

• DO NOT RUN. (step 2.2) Check if we can run script.

Proposed Spec (RUN)

• RUN. The check is removed.

Chromium implementation (RUN)

Microtasks

HostEnqueuePromiseJob

Current Spec (DO NOT RUN)

• DO NOT RUN. (step 2.2). Check if we can run script.

Proposed Spec (RUN)

• RUN. The check is removed.

Chromium implementation (RUN)

[zcorpan]

Thanks @sadym-chromium, @juliandescottes. I think it's now clear that it's intentional to run some scripts and why.