Update WAVE test runner

[FritzHeiden]

trafficstars

This PR updates the wave test runner used in WMAS and DPCTF projects by CTA. It mainly includes bug fixes as well as change to support python 3.8

[foolip]

@FritzHeiden is there someone familiar with this setup who you'd like review from? If not, then as long as the CI passes and the diff looks reasonable, someone like me could rubberstamp and merge it. Right now the CI isn't passing, have you checked that out?

[JohnRiv]

@foolip as the WAVE HATF chair, I'll review this PR for the implementation. Agree we need tests to pass as well for this to be able to be merged.

[FritzHeiden]

Unfortunately, I don't know how to resolve these errors:

[Errno 2] No such file or directory: '/home/test/web-platform-tests/tools/wave/../wptrunner/requirements_chrome.txt'

This is not part of the tools/wave directory, so I don't know what this is for and how to resolve it. Can someone please assist me with this?

[louaybassbouss]

Thanks, @FritzHeiden, for your contribution. @JohnRiv, can you please review? It seems that the new failures in the pipeline are not related to wave, and there are no indications in the logs regarding the reason for the failures.

[JohnRiv]

Thanks @FritzHeiden & @louaybassbouss. Here is what I can gather from the latest failures:

There are essentially 2 things failing (that manifest as 5 failing checks):

1. ERROR WebDriver was not accessible within 30 seconds. / CRITICAL Max restarts exceeded

• See Azure Pipelines (infrastructure/ tests: macOS)
• This seems unrelated to the changes, and hopefully when you push up a new PR, this will resolve itself 🤞

2. no tests run when attempting to run pytest in /tools/wave

• See Community-TC Integration / tools/ integration tests (Python 3.8) & Community-TC Integration / tools/ integration tests (Python 3.12)
• This happens because as part of dda2a3d, the tests in tools/wave/tests/test_wave.py were commented out. I assume if you fix that, the commented out tests in tools/wave/tests/test_api.py will then error, so either fix those tests, or if they truly are not needed, then remove them and the task to run them

[FritzHeiden]

@JohnRiv Thanks for you help. I removed the two test files and the error still occurs. Do you have another idea how to prevent the pipeline from trying to run tests in the wave directory?

[JohnRiv]

@FritzHeiden it appears this is what executes the command to run those tests:

https://github.com/web-platform-tests/wpt/blob/master/tools/ci/ci_tools_integration_test.sh#L16-L20

so that code will have to be removed as well (or the tests should be fixed)

[FritzHeiden]

@JohnRiv Removing these lines resolves the issues from the CI. The other two failing checks don't seem to be related to the changes in this PR.

[JohnRiv]

@FritzHeiden can you try pulling in the latest from the master branch to see if that either fixes the failure, or provides a more useful understanding of the issue?

[JohnRiv]

Thanks @FritzHeiden. The failures I see in the logs are ones I see in other PRs, so it definitely seems unrelated to the changes here:

web-platform-test
~~~~~~~~~~~~~~~~~
Ran 379 checks (214 subtests, 165 tests)
Expected results: 366
Skipped: 8 tests
Unexpected results: 5
  test: 5 (5 crash)

Unexpected Results
------------------
/infrastructure/assumptions/canvas-background.html
  CRASH /infrastructure/assumptions/canvas-background.html
/infrastructure/reftest/reftest.www.sub.html
  CRASH /infrastructure/reftest/reftest.www.sub.html
/infrastructure/reftest/reftest_ref_timeout.html
  CRASH /infrastructure/reftest/reftest_ref_timeout.html
/infrastructure/reftest/reftest_timeout.html
  CRASH /infrastructure/reftest/reftest_timeout.html
/infrastructure/crashtests/testdriver.html
  CRASH /infrastructure/crashtests/testdriver.html

[JohnRiv]

@foolip @jgraham @DanielRyanSmith, these changes are good from WAVE's perspective. Do you have any requested changes, and if not, can this be merged in?

[JohnRiv]

Here is a copy of the full log from the failing Run tests (Safari Technology Preview) job in case it is no longer available when someone is available to review this PR: 20250320.18.191.SafariTechnologyPreview.log

[louaybassbouss]

@foolip @jgraham @DanielRyanSmith please let us know if anything you need from us (@FritzHeiden and me).