results-analysis

results-analysis copied to clipboard

If any non-OK test status occurs in any labeled test that is being scored for interoperability, and that test was not added to the list of non-OK statuses to ignore in the code, the entirety of the score update GitHub Action will fail, and interop scores will not be updated unless the test is either investigated and explicitly added to to the list of ignored tests statuses.

Because the results are checked for all major parallel browser runs over time, any test that has ever encountered a non-OK test status will need to be added to this list or the interop score will no longer update. This includes transient errors like the one for /css/css-scroll-snap/snap-at-user-scroll-end.html. This test incurred a non-OK test status (timeout) on this run, but has not seen another non-OK status afterward. However, the entirety of Interop scoring will be halted until this test is permanently added to the list of known non-OK test statuses indefinitely.

FYI @foolip @jgraham

I think it would make sense to treat this as a review/triage problem. For Interop 2022, I put together https://github.com/web-platform-tests/interop/issues/48, and that resulted in filing at least two bugs. After that point, I don't think there were any new non-OK harness statuses, /css/css-scroll-snap/snap-at-user-scroll-end.html is the first.

The initial triage is useful, and I think what we should do is look at searches like these, using Interop 2022 for illustration purposes: https://wpt.fyi/results/?label=master&label=experimental&product=chrome&product=firefox&product=safari&aligned&view=interop&q=status%3Aerror%20and%20%28label%3Ainterop-2021-aspect-ratio%20or%20label%3Ainterop-2021-flexbox%20or%20label%3Ainterop-2021-grid%20or%20label%3Ainterop-2021-position-sticky%20or%20label%3Ainterop-2021-transforms%20or%20label%3Ainterop-2022-cascade%20or%20label%3Ainterop-2022-color%20or%20label%3Ainterop-2022-contain%20or%20label%3Ainterop-2022-dialog%20or%20label%3Ainterop-2022-forms%20or%20label%3Ainterop-2022-scrolling%20or%20label%3Ainterop-2022-subgrid%20or%20label%3Ainterop-2022-text%20or%20label%3Ainterop-2022-viewport%20or%20label%3Ainterop-2022-webcompat%29

note we now haven't had any updated stats for over a week due to this

Fixing: https://github.com/web-platform-tests/wpt/pull/38387 https://github.com/web-platform-tests/results-analysis/pull/146

But I also think our setup is too fragile, it's not the right tradeoff for all Interop 2023 updates to stop and have to be investigated/fixed by the interop team because of a regression in one focus area. I'll send a PR with what I think we should do. I'll create an issue in the interop repo to discuss.