hail
hail copied to clipboard
[benchmark] run benchmark suite on release
We want to track Hail's performance with every release for a number of reasons, including but not limited to:
- Measure how we are doing in delivering value to scientists
- Measure the effect of changes, test our intuition and learn how to improve the product.
- Compare our solution with others
- Catch unexpected regressions
As of the time of writing, benchmarks are run rarely and have rotted somewhat. There's a bit of work required to get them going again. There's also some work in getting them running in CI and capturing the results. Very roughly, I think work can broken down as follows:
- [ ] get benchmarks passing
- [ ] organise trials with learnings from https://www.zora.uzh.ch/id/eprint/170445/1/emse_smb_cloud.pdf
- [x] run bechmarks in ci on deploy and store the results somewhere appropriate, fail if there's something really awful
- [ ] visualise results on some appropriate cadance for trends. Might be nice to have a graphic on our github page.
I think many of these can be done in parallel.