interop
interop copied to clipboard
web-platform-tests
Avoid unintended scope changes to active focus areas
Maybe we should have a CI check (in wpt) to avoid test changes for tests in interop-* (for active focus areas). Require a commit message to link to an issue in the interop repo?
c.f. https://github.com/web-platform-tests/interop/issues/345 , https://github.com/web-platform-tests/interop/issues/338
@zcorpan any thoughts on how to implement this? The labels are in https://github.com/web-platform-tests/wpt-metadata and can be accessed via a wpt.fyi API. Either would be an external dependency which would increase the risk of CI failures if those dependencies are not reliable.
I had in mind doing a git clone of the wpt-metadata repo. I'm not so familiar with the wpt.fyi API. No opinion on which approach to use. If there are network issues we could have a retry (or several retries with increased timeout).
Looks like the API is undocumented (https://github.com/web-platform-tests/wpt.fyi/issues/3337) but one can get all of the labels in a single API call at https://wpt.fyi/api/metadata?product=chrome&includeTestLevel.
Whether using that API or a clone of the wpt-metadata repo, we'd get a list of test names. I'm a bit unsure how to implement the rest of the check, but it is possible of course.
I think there's code to determine "affected tests" for a PR for running stability checks, which maybe we could reuse? cc @jgraham
We discussed this in https://github.com/web-platform-tests/interop/issues/660. Concrete next steps here could be a GitHub Actions workflow that does:
- Get list of labels for active focus areas from https://wpt.fyi/static/interop-data.json
- Get the list of labeled tests from https://wpt.fyi/api/metadata?includeTestLevel=true&product=chrome
- Get affected tests with
./wpt tests-affected(this also includes tests affected indirectly through included scripts) - Check if there's an intersection between the labeled tests and the affected tests
- Add a "affects-interop-2024" label to the pull request