wpt

wpt copied to clipboard

In https://github.com/web-platform-tests/interop-2022/issues/96 I found differences between Chrome and Edge in the form of missing results from Edge. Here are two runs from the same commit of wpt:

• Chrome with 47446 tests: https://wpt.fyi/results/?run_id=5762866111053824
• Edge with 44617 tests: https://wpt.fyi/results/?&run_id=5668150002581504

That's 2829 tests fewer. @mustjab have you seen such differences in the past, is it a known issue? Incomplete results will make it difficult to use the Edge results for spotting differences to Chrome, and other uses.

Taking another pair of runs, from the previous commit:

• Chrome with 47446 tests: https://wpt.fyi/results/?run_id=5728025067913216
• Edge with 46711 tests: https://wpt.fyi/results/?run_id=5758360556142592

The Chrome number is the same but the Edge number is not. This suggests there's some flakiness involved.

We've seen differences between Chrome and Edge runs, a lot of these differences are due to diff between runs on Linux vs runs on Windows. Don't believe we ever investigated the reason behind these differences.

Are we able to run Chrome runs on Windows for comparison?