criterion.rs
criterion.rs copied to clipboard
bheisler
Idea: Relative Time
In an effort to have more stable benchmarks results, do you think having a baseline function (internal to Criterion) e.g. sqrt in a loop that is run during warmup and report the benchmark results relative to that could help?
As an example, let's imagine that running various iterations of sqrt on an idle CPU takes 2 µs on average. The user wants to benchmark fibonacci(20) which takes 20 µs on the same idle CPU. Criterion would report a relative_time of 10 µs (20/2).
My hypothesis is that on a busier CPU the baseline sqrt would take more time and thus will fibonacci(20).
Maybe this is something that other projects have explored but wanted to hear your opinion.
I doubt this would give more stable results. What's your end goal? Stable results even when the CPU is under load? I'm not sure that is possible.
What's your end goal? Stable results even when the CPU is under load?
Yes, this is the goal. I have a very cheap way to verify if this would work without changing Criterion code. That is, by just adding a fibonacci to my benchmark and compute the relative time manually over some runs with / without CPU load.