helios icon indicating copy to clipboard operation
helios copied to clipboard

feat: Criterion + Iai Benchmarking [RFC]

Open refcell opened this issue 2 years ago • 2 comments

Overview

Introduces criterion statistical benchmarking as well as iai one-shot cachegrind benchmarking.

refcell avatar Dec 06 '22 17:12 refcell

This is really cool. Didn't know about the criterion crate.

We probably want to add the results of the benchmarks somewhere as well.

ncitron avatar Dec 06 '22 18:12 ncitron

Probably worth benchmarking at a lower level (skipping network) in addition to the integration benches.

I would suggest a bench of a receipt proof + storage proof. Not a blocker on this PR though.

sragss avatar Dec 07 '22 16:12 sragss

Probably worth benchmarking at a lower level (skipping network) in addition to the integration benches.

I would suggest a bench of a receipt proof + storage proof. Not a blocker on this PR though.

Would you be willing to spec this out in an issue so I (or some other interested party) can implement in a diff pr?

refcell avatar Dec 11 '22 00:12 refcell

I'm also not sure if we wan't to exclude the network from the bechmark. Although we have no control of it, the network is the slowest part of every call. I was thinking the benchmarks would just give a sense of how long Helios takes to do most things, rather than some theoretical "how fast is the code executing" metric.

ncitron avatar Dec 11 '22 00:12 ncitron

ye, taking offline so we can discuss best way forward

refcell avatar Dec 11 '22 00:12 refcell