node
node copied to clipboard
nodejs
test: update WPT runner
This PR updates the WPT runner in an attempt to fix the following issues:
- if a test file has a single expected failure new regressions are not caught
- if a test file has expected failures and doesn't fail the runner does nothing with the information
- if a test file has a flaky test the only possibility is to skip the whole file
In the following way:
- expected failures are now test names, not arbitrary strings
- all expected failures must be present in the results otherwise the runner fails with an
unexpected pass - a
flakyarray of test names can also be present to allow tests files to be executed whilst containing individual flaky tests.
Other updates (e.g. to the status files, or wpt entry points)
- "skip" instead of "fail" to reflect what the actual behaviour was
- updating expected failures to be the reported failed test names rather than arbitrary descriptions
- allowing more tests that weren't run before by adding a simple polyfill to fix the
location is not definederror. This one is coming from wpt orchestration (test/fixtures/wpt/common/subset-tests.js), not the actual tests. - CI worker concurrency management that would bog the tests down due to the unlocked WebCryptoAPI WPT tests.
cc @joyeecheung, she did the last refactor (#33297) to the runner behaviour v status file format.
I am unsure which teams to tag here.
I think it should cc to @nodejs/testing
I think I'll further refactor to allow the expected failures be the test names, not just the errors. This will avoid issues where the error includes a cwd (like the current CI failures).
I also feel hard to change something related to web compatibility since regressions are not detected with the current runner. Hope that this PR gets progressed.
Needs investigating:
-
ubuntu1804_sharedlibs_withoutintl_x64has an unexpected pass the othertest-commit-linux-containeredbuild types don't (intest.wpt/test-encoding) -
win2012r2-COMPILED_BY-vs2019-x86has an unexpected failure the other build types don't (intest.wpt/test-broadcastchannel)
Not sure how to build in overrides for specific platforms that would allow to target this.
ubuntu1804_sharedlibs_withoutintl_x64has an unexpected pass the othertest-commit-linux-containeredbuild types don't (intest.wpt/test-encoding)
This is because the file is conditionally skipped
---- streams/decode-utf8.any.js ----
[SKIPPED] requires small-icu
I need to account for such scenario.
win2012r2-COMPILED_BY-vs2019-x86has an unexpected failure the other build types don't (intest.wpt/test-broadcastchannel)
may be flaky
Are the without-intl and without-ssl failures for wpt-console related?
They used to be silently ignored before. Now they're out in the light.
I'll flake the tests or skip the file (only 2 options we have) and open an issue for console to investigate.