Automate testing that http://web-platform.test:8000/tools/runner/index.html works and runs tests as expected

[sideshowbarker]

We should have a (infra) test that checks http://web-platform.test:8000/tools/runner/index.html to make sure it actually runs wpt tests without throwing or hanging.

Context: We have evidence that there are a number of groups working on specs who rely on the web-based runner as part of their workflow. Even though we encourage them to move to using the command-line runner, and give them how-to guidance on using it, we’re still going to have people and groups choosing to use the web-based runner — at least as long as we continue to provide it.

So if we are going to continue to provide it, we should be doing automated testing to ensure its basic operations are working as expected — including that it’s parsing and loading the manifest as expected, and that it’s actually capable of running a test and reporting results for it. So essentially just a basic smoke test.

Otherwise, lacking any infrastructure in place to make sure it isn’t broken, we will continue breaking it — because we’re not using it ourselves and are otherwise not getting any indication that some change we’ve made to the backend hasn’t regressed its operation.

[mosaadm]

@sideshowbarker Just to make sure I am doing the right solution. It should do the following if I understood your request:

1. ./wpt serve
2. Open a browser
3. Type some path for tests ex. '/dom/abort/'
4. Click start
5. Check results Also I assume this should run in CI. Should it be a seperate job ? and should it be GH actions or something else?

[sideshowbarker]

@sideshowbarker Just to make sure I am doing the right solution. It should do the following if I understood your request:

1. ./wpt serve

2. Open a browser

3. Type some path for tests ex. '/dom/abort/'

4. Click start

5. Check results

Yup

Also I assume this should run in CI.

Yes

Should it be a seperate job ?

Yes

and should it be GH actions or something else?

GH Actions