Feature request: report results in terms of passing tests (not subtests)

[mathiasbynens]

trafficstars

My use case: Throughout the year, I’d like to easily track the progress towards my team’s goal of increasing the test pass rate for a specific set of tests, for the specific browser we are working on.

Illustrative example: For the specific scenario I’m talking about, I’ve created the chromium-bidi-2023 label which lets us view the results of the desired set of tests: https://wpt.fyi/results/webdriver/tests/bidi?q=label:chromium-bidi-2023 However, what’s missing is an easy & stable way to interpret the test pass rate. The UI currently looks like this:

It says things like “Subtest Total: 2667 / 2911” while at the same time the blue box at the top says “Showing 174 tests (3232 subtests)”. As a user, it’s unclear whether 2911 or 3232 is the total number of subtests. As it turns out, neither number is guaranteed to be correct, since even the larger number (in the blue box) only counts subtests that have run overall (across any of the browsers) — it’d still miss any subtests that time out on all browsers, for example. This makes it hard to use these metrics for progress tracking.

One possible solution could be to show total tests instead of subtests, e.g. “145 / 174”, which would be more stable — but I’m open to other ideas.

[foolip]

This is something that I've often wanted as well, since I often want to know where there are the most tests with some failure, and subtest counts are not a great guide to that. I think a view=test that counts fully passing tests would be very useful.

[foolip]

As part of this, I think it's also a great opportunity to inline the test statuses to not have to click through, similar to what I suggested for reftests previously: https://github.com/web-platform-tests/wpt.fyi/issues/421

[foolip]

@jcscottiii has implemented something for this now and there's an RFC up at https://github.com/web-platform-tests/rfcs/pull/190. @mathiasbynens can you try it out?

[mathiasbynens]

The staging URLs:

• https://staging.wpt.fyi/results/webdriver/tests/bidi?label=experimental&product=chrome&view=test&q=label%3Achromium-bidi-2023 (with label filtering)
• https://staging.wpt.fyi/results/webdriver/tests/bidi?label=experimental&product=chrome&view=test (without label filtering)

…solve our use case perfectly. Thanks so much for prototyping this!