bugbug
bugbug copied to clipboard
Add some WPT-related stats to the team-level view
Maybe in the same chart where we show the skip-if data:
- # of disabled tests (maybe add it up to the number of tests with skip-if conditions?)
- # of "expected crash" tests
- # of tests that uniquely fail on Firefox
The first two can be retrieved from gecko.v2.mozilla-central.latest.source.source-wpt-metadata-summary tasks (summary.json artifact), e.g. https://firefoxci.taskcluster-artifacts.net/HYkJQqHdRQmf3_cnMDu6Jg/0/public/summary.json.
The last can be retrieved by using the wpt.fyi API.
For the meta summary file, the format is something like
{
dir_name:
"_tests": {
test_name: {
property: [[condition, value]]
"_subtests": {
name: {
property: [[condition, value]]
}
}
}
},
property: [[condition, value]]
}
Where property is something like disabled, intermittent, lsan-allowed, expected_CRASH, etc. condition is a string serilization of the conditional that is evaluated against mozinfo associated with the given value. Data can be on the directory, test, or subtest level; obviously not eveything makes sense in all those cases e.g. crashes apply only to whole tests and lsan failures apply either to whole tests or whole directories. Only tests/subtests with "interesting" properties are included. https://searchfox.org/mozilla-central/source/testing/web-platform/metasummary.py is the code that extracts this from the wpt expectation ini files.
For the wpt.fyi data, https://github.com/web-platform-tests/wpt.fyi/blob/main/api/README.md has some documentation, and you should also be able to copy the requests from https://jgraham.github.io/wptdash/ I don't know if we are OK with querying that all the time, or want some local caching of numbers.