results-analysis icon indicating copy to clipboard operation
results-analysis copied to clipboard

New runs between running git-write.js and one of the scoring scripts will fail

Open foolip opened this issue 2 years ago • 2 comments
trafficstars

The problem is that first we do this to populate the most recent runs:

https://github.com/web-platform-tests/results-analysis/blob/e16852115077721b680c5a3e1a9a99b7ac66fb4f/build.sh#L17

The in various scoring scripts, we use the wpt.fyi API to enumerate runs:

https://github.com/web-platform-tests/results-analysis/blob/e16852115077721b680c5a3e1a9a99b7ac66fb4f/interop-scoring/main.js#L405-L406

And then fail if any of those aren't in the "cache":

https://github.com/web-platform-tests/results-analysis/blob/e16852115077721b680c5a3e1a9a99b7ac66fb4f/interop-scoring/main.js#L419-L424

This happened in https://github.com/web-platform-tests/results-analysis/actions/runs/4240081460/jobs/7368750202

This doesn't happen often, but it will happen occasionally.

A better setup would be one of:

  • Assume the cache is sufficiently up-to-date, don't try to update it, and ignore any run not in it
  • Don't run git-write.js at all, and instead handle cache misses by creating the tree from the full wptreport.json. This would work without the results-analysis-cache repo, just very slowly.
  • Get the runs from the wpt.fyi API once, ensure those are in the cache, and then run scoring scripts (storing the runs in some intermediate JSON file)

foolip avatar Feb 22 '23 09:02 foolip

Assume the cache is sufficiently up-to-date, don't try to update it, and ignore any run not in it

If we want to do this, we probably want to do substantially increase the frequency of the runs. But especially around midnight UTC there can be long delay of GitHub Actions jobs, which means even if the cache update job was hourly there would be a decent chance that we'd run the scoring update job before the cache job.

gsnedders avatar Feb 22 '23 19:02 gsnedders

It happened again in https://github.com/web-platform-tests/results-analysis/actions/runs/4475248771/jobs/7864482662

foolip avatar Mar 21 '23 07:03 foolip