hail icon indicating copy to clipboard operation
hail copied to clipboard

[benchmark] run benchmark suite on release

Open ehigham opened this issue 5 months ago • 0 comments

We want to track Hail's performance with every release for a number of reasons, including but not limited to:

  • Measure how we are doing in delivering value to scientists
  • Measure the effect of changes, test our intuition and learn how to improve the product.
  • Compare our solution with others
  • Catch unexpected regressions

As of the time of writing, benchmarks are run rarely and have rotted somewhat. There's a bit of work required to get them going again. There's also some work in getting them running in CI and capturing the results. Very roughly, I think work can broken down as follows:

  • [ ] get benchmarks passing
  • [ ] organise trials with learnings from https://www.zora.uzh.ch/id/eprint/170445/1/emse_smb_cloud.pdf
  • [x] run bechmarks in ci on deploy and store the results somewhere appropriate, fail if there's something really awful
  • [ ] visualise results on some appropriate cadance for trends. Might be nice to have a graphic on our github page.

I think many of these can be done in parallel.

ehigham avatar Jan 30 '24 19:01 ehigham