minitrace-rust icon indicating copy to clipboard operation
minitrace-rust copied to clipboard

Published 20 hours ago • verified publisher icon tikv

Continuous benchmark

Open breezewish opened this issue 3 years ago • 3 comments

As we care performance a lot, it may be necessary to run benchmarks for each PR submission.

breezewish avatar Mar 03 '22 14:03 breezewish

We could try utilizing GitHub Action for continuous benchmarking, by comparing to relative regressions. Ref https://github.com/ethereumjs/ethereumjs-monorepo/issues/897

breezewish avatar Mar 04 '22 03:03 breezewish

High priority IMO. Regressions are devilishly difficult to identify after the fact. The criterion author had a crate targeting CI use cases. Let me try to dig it up.

taqtiqa-mark avatar Mar 15 '22 11:03 taqtiqa-mark

IMO the approach to take is to create benchmarks using iai rather than criterion - not that you can't do both. However, I believe if you want to accept/reject/evaluate the performance impact of a PR using github actions, or some such tooling, you need to abstract the hardware/virtualware which leaves one looking at iai.

Not sure if there are alternatives.

taqtiqa-mark avatar Mar 15 '22 11:03 taqtiqa-mark