current-bench
current-bench copied to clipboard
Github bot messages
It's easy to forget checking the benchmarks on each PR, because it's generally not necessary.
Current-Bench could detect when a metric moves by more than 10% (?) and send a warning message to the PR: "You probably want to check those metrics!" For inspiration, see an example codecov report with pretty graphs embeded (but I think we should aim for a shorter message than this!)
Both regressions and improvements should be reported: a benchmark improvement can actually be indicative of a bug where real work has been skipped! And it's otherwise a nice confirmation that the PR fixed a performance issue :)
See also #143 though this seems nicer to me.