wpt.fyi

wpt.fyi copied to clipboard

In https://github.com/web-platform-tests/wpt/pull/18790, https://tools.taskcluster.net/groups/Fc3IduRARwiva4bgSkPfOA looks like this now:

I've had the experience quite often of waiting for the Chrome/Firefox results on a wpt PR, and it not arriving until quite a bit after the Safari results. I think this is because wpt.fyi waits until Taskcluster as a whole has finished, but I haven't verified in code.

If my suspicion is correct, then fixing https://github.com/web-platform-tests/wpt.fyi/issues/1240 might be a part of fixing this, because if wpt switches to the checks integration we would presumably see each task as an individual check, making it straightforward to react to each individual check instead of the overall status as we currently do.

@Hexcles @lukebjerring

https://github.com/web-platform-tests/wpt/pull/18832 is another case of this, where lots of tests are affected and I expect stability jobs to time out or possibly fail due to pre-existing flakiness. I'll have to wait much longer than it takes to run the tests to know what the changes did to the Chrome results.

I had this problem again today. https://github.com/web-platform-tests/wpt/pull/23341 is taking a very long time to run stability jobs, and as a results I couldn't check the results for https://chromium-review.googlesource.com/c/chromium/src/+/2172971 but will have to come back to it later.

We've migrated our Taskcluster integration to use GitHub Checks, which means we now get a check run (and corresponding events) for each task (example). This should give us the ability to digest results much faster (as soon as any browser completes) instead of waiting for everything to complete.

That's great! I guess that right now we wait for the overall check suite to finish, but that it's not actually simpler than waiting for the check directly, since we have to know about the check anyway?

There's the minor difference that we need both the base and head run to finish to report results, which is a given when we get the check suite complete event. So that's a non-trivial change in logic, we will have to track what we've previously seen for a given PR, making sure to handle re-requests correctly too I think.

I've been sending some PRs today and this is really noticeable. Safari results are produced very fast, but I have to wait for a long time for Chrome and Firefox. (In https://github.com/web-platform-tests/wpt/pull/26256.)