results-analysis
results-analysis copied to clipboard
New runs between running git-write.js and one of the scoring scripts will fail
The problem is that first we do this to populate the most recent runs:
https://github.com/web-platform-tests/results-analysis/blob/e16852115077721b680c5a3e1a9a99b7ac66fb4f/build.sh#L17
The in various scoring scripts, we use the wpt.fyi API to enumerate runs:
https://github.com/web-platform-tests/results-analysis/blob/e16852115077721b680c5a3e1a9a99b7ac66fb4f/interop-scoring/main.js#L405-L406
And then fail if any of those aren't in the "cache":
https://github.com/web-platform-tests/results-analysis/blob/e16852115077721b680c5a3e1a9a99b7ac66fb4f/interop-scoring/main.js#L419-L424
This happened in https://github.com/web-platform-tests/results-analysis/actions/runs/4240081460/jobs/7368750202
This doesn't happen often, but it will happen occasionally.
A better setup would be one of:
- Assume the cache is sufficiently up-to-date, don't try to update it, and ignore any run not in it
- Don't run git-write.js at all, and instead handle cache misses by creating the tree from the full wptreport.json. This would work without the results-analysis-cache repo, just very slowly.
- Get the runs from the wpt.fyi API once, ensure those are in the cache, and then run scoring scripts (storing the runs in some intermediate JSON file)
Assume the cache is sufficiently up-to-date, don't try to update it, and ignore any run not in it
If we want to do this, we probably want to do substantially increase the frequency of the runs. But especially around midnight UTC there can be long delay of GitHub Actions jobs, which means even if the cache update job was hourly there would be a decent chance that we'd run the scoring update job before the cache job.
It happened again in https://github.com/web-platform-tests/results-analysis/actions/runs/4475248771/jobs/7864482662