criterion.rs
criterion.rs copied to clipboard
Allow `.iter_custom()` to return multiple timings
I've got a multi-threaded benchmark running within an .iter_custom()
call and I'd rather be able to return each thread's execution time instead of the averaged time across all threads, since that would give a more detailed picture of the data
Hello! Thanks for the suggestion and for the patience.
I'm not opposed to the idea, but it would require a lot of design and implementation work. The assumption that each measurement produces one number is pretty deeply baked into Criterion-rs. How should multiple measurements per benchmark be analyzed? How should they be reported, on the CLI, in the HTML reports, in the various machine-readable output formats?
I don't really remember the context of what I wanted this for, but I believe I wanted to just hand criterion my raw benchmark results in order to enable it to average/analyze all of the datapoints so that .iter_custom() -> Vec<1000 results>
would be analyzed properly and display outliers, percentiles, etc.