web-platform-tests
Web runner test timeouts and manual test issues
Background
I was asked to run the web-platform-tests against some of the DuckDuckGo browsers recently, and had some difficulty. The browsers unfortunately don't apparently support WebDriver yet and so I was stuck using the web runner. I realise these issues are probably not new, but I thought it might be worth writing up my findings in case they help.
Timeouts
The main issue I ran into was with test timeouts. After some trial and error, it seemed that certain tests somehow clash with the web runner and cause all further tests fail to run and timeout. If those tests are avoided using the "Run tests under path" field, the later tests are run OK. That didn't seem to be browser dependent, I could reproduce on Edge, Safari and two of the DuckDuckGo browsers.
By running the tests and looking back to the first timeout, I could guess at which tests are problematic for the web runner. These could all be wrong, perhaps they just happened to timeout before a problematic test for example, but perhaps listing them here will help save someone time:
-
/IndexedDB/serialize-sharedarraybuffer-throws.https.html -
/content-security-policy/inheritance/blob-inherits-from-meta-http-equiv-with-invalid-characters.html -
/css/css-grid/subgrid/subgrid-baseline-010.html -
/css/css-masking/parsing/clip-path-computed.html -
/direct-sockets/disabled-by-permissions-policy.https.sub.html -
/encoding/sharedarraybuffer.https.html -
/fenced-frame/embedder-coop-coep-blocked.https.html -
/html/cross-origin-embedder-policy/credentialless/script.https.window.html -
/html/browsers/origin/origin-keyed-agent-clusters/getter-special-cases/cross-origin-isolated.sub.https.html -
/html/browsers/browsing-the-web/unloading-documents/prompt-and-unload-script-uncloseable.html -
/hr-time/clamped-time-origin-isolated.https.html -
/html/browsers/browsing-the-web/unloading-documents/prompt-and-unload-script-uncloseable.html -
/js-self-profiling/tentative/marker-vm-state.https.html -
/measure-memory/detached.https.window.html -
/scroll-animations/view-timelines/fieldset-source.html -
/webaudio/the-audio-api/the-audioworklet-interface/audioworklet-postmessage-sharedarraybuffer.https.html -
/webcodecs/audio-data.crossOriginIsolated.https.any.html -
/workers/postMessage_DataCloneErr.htm
Suggestion: If having the web runner handle all these test cases isn't possible, perhaps there should be a way to mark such problematic tests so that they are skipped by the web runner?
Manual tests
When using the web runner, I unchecked the "Manual Tests" option, but I still found a lot of tests that require manual verification ran. That required me to click "Not Run" over and over again at the end of my test run. If it helps, the dialogues said something like this"Manual Testing - The current test requires manual result marking. Test and ref should compare equal".
Suggestion: Unchecking manual tests should prevent such tests from running.
Test manifest
Sometimes I found the web runner gets stuck on "Updating and loading test manifest; this may take several minutes". Sometimes that finishes almost instantly, sometimes it does take a few minutes, but sometimes it will get stuck forever. This is quite frustrating when doing a big test run, sometimes you'll check on the run hours later and found it never started.
Workaround
In the end, I used the "Use regular expression" option and had a crazy test filter like this to skip as many of the problematic timeout tests and manual tests as I could: ^\/(?!IndexedDB)(?!content-security-policy)(?!css)(?!direct-sockets)(?!encoding)(?!fenced-frame)(?!hr-time)(?!html)(?!js-self-profiling)(?!measure-memory)(?!mathml)(?!navigation-api)(?!png)(?!scroll-animations)(?!svg)(?!webaudio)(?!webcodecs)(?!webvtt)(?!workers\/postMessage).*
This took ages to come up with, I did repeated separate test runs for each letter to figure out which tests would cause runs to start timing out. I wouldn't wish that experience on folks! It also means that a lot of tests that can run fine are skipped.
I didn't find a proper workaround for the test manifest issue, instead I tried to keep checking on the test runs, and sometimes restarting them, until I saw that the test manifest update had finished.
Thanks for raising this issue. It’s good to have a record here of more of the known issues — if for no other reason, so that we have something we can point others too later.
Suggestion: If having the web runner handle all these test cases isn't possible, perhaps there should be a way to mark such problematic tests so that they are skipped by the web runner?
Unfortunately, given the lack of anybody actively maintaining the runner itself for 5 years or more now, it’s very unlikely that anybody new is going to step forward to implement any changes related to the behavior of the runner — and even somewhat unlikely that any reviewers from the project would be interested in making time to review a patch from anyone.
Other than general changes to the underlying Python stuff that the runner relies on along with other parts of the WPT code, nobody has been putting any work into the runner code for many years now.
Using instead the instructions at https://web-platform-tests.org/running-tests/from-local-system.html to run tests from the shell using wpt run continues to be the only recommended and reliable way to run tests and ensure you’re actually getting accurate results.
