criterion.rs icon indicating copy to clipboard operation
criterion.rs copied to clipboard

Allow `.iter_custom()` to return multiple timings

Open Kixiron opened this issue 2 years ago • 2 comments

I've got a multi-threaded benchmark running within an .iter_custom() call and I'd rather be able to return each thread's execution time instead of the averaged time across all threads, since that would give a more detailed picture of the data

Kixiron avatar May 02 '22 14:05 Kixiron

Hello! Thanks for the suggestion and for the patience.

I'm not opposed to the idea, but it would require a lot of design and implementation work. The assumption that each measurement produces one number is pretty deeply baked into Criterion-rs. How should multiple measurements per benchmark be analyzed? How should they be reported, on the CLI, in the HTML reports, in the various machine-readable output formats?

bheisler avatar Jul 14 '22 23:07 bheisler

I don't really remember the context of what I wanted this for, but I believe I wanted to just hand criterion my raw benchmark results in order to enable it to average/analyze all of the datapoints so that .iter_custom() -> Vec<1000 results> would be analyzed properly and display outliers, percentiles, etc.

Kixiron avatar Jul 15 '22 18:07 Kixiron