cargo-criterion icon indicating copy to clipboard operation
cargo-criterion copied to clipboard

Published 20 hours ago • verified publisher icon bheisler

Add CDF plot to benchmark reports

Open bheisler opened this issue 4 years ago • 2 comments

CDF plots are less commonly understood, but once one learns to read them they're actually much better at conveying information than the PDF plots cargo-criterion generates. We should generate a CDF plot as well as the PDF plot for the more advanced users.

Could maybe generate a normal-quantile plot for the really advanced users as well?

bheisler avatar Jan 14 '21 02:01 bheisler

Related issue in criterion.rs: https://github.com/bheisler/criterion.rs/issues/443

link2xt avatar Jan 14 '21 05:01 link2xt

See top plot at https://github.com/deltachat/deltachat-core-rust/pull/2382#issuecomment-826385834 for an example of how misleading PDF plots can be. Red curve suggests some values fall below 300ms, but in fact there are no runs at all which completed in under 300ms.

link2xt avatar Apr 25 '21 20:04 link2xt