wpt.fyi
wpt.fyi copied to clipboard
web-platform-tests
When moving/adding tests between runs summary tables show same test count for before/after
e.g. in https://wpt.fyi/results/css/selectors?diff&filter=ADC&run_id=5719367384825856&run_id=5192487070072832 where all the tests were moved into the new location. The summary table shows the same number of tests for the "before" and "after" runs, but when you drill down into the individual tests, it (correctly) shows that all the tests in the before run have a status of MISSING.
What I wanted to do in this case was check that the PR (which just moved tests) resulted in the same tests passing/failing after the move, but that turned out to be hard since the pre-move tests are filtered out, and even if they weren't you'd have to look at each individual test to work out if the results matched.
I think there are multiple issues here. One is what we should do with missing tests in the general case. We currently show those as "0 / 1" when summarized, but we should probably count those as "0 / 0".
What I think you're noticing here is the rename detection that happens for PR diffs. /api/diff internally uses the GitHub API to detect renames, and then there's some code that merges things in the UI to make it easier to compare the before/after. Sounds like you think it made it harder?
You can actually see the rename detection kick in if you reload the page. It happens async so first you get a glimpse of the original state of things.
@jgraham if I were looking for regressions here I'd just note that the numbers in https://wpt.fyi/results/css?diff&filter=ADC&run_id=5719367384825856&run_id=5192487070072832 are the same. It looks like checking it in detail is actually hard, because queries like is:different and seq(status:pass status:!pass) happen on the server side and the rename detection is on the client side...
Oh, so it's transparently handling the rename on the summary pages, but not making that explicit, or doing the same on individual test pages? That explains why I was so confused. Knowing that's the behaviour I now understand what I was seeing, but until you explained it the UI was inscrutable.
Maybe having some way to see that the file was detected as renamed in the summary views would work (but that would still make it confusing that the test details pages work differently).