criterion.rs icon indicating copy to clipboard operation
criterion.rs copied to clipboard

Published 20 hours ago • verified publisher icon bheisler

Criterion version 0.4 and 0.5 milestones

Open lemmih opened this issue 2 years ago • 6 comments

What should be in criterion-0.4 and criterion-0.5? Few things are set in stone so submit your comments and ideas.

Version 0.4:

  • [x] Remove all deprecated functions.
  • [x] Make CSV output an optional and non-default feature.
  • [x] Make HTML output an optional and non-default feature.
  • [x] Make rayon an optional and default feature. Rayon improves the analysis performance but isn't required for correctness.
  • [x] Make plotters an optional and non-default feature.
  • [x] Print status messages to stderr, benchmark results to stdout.
  • [x] Allow durations from command-line arguments to be fractions of a second. For example, --warm-up-time=0.5.
  • [x] Formally support WASI.

Version 0.5:

  • [ ] Remove CSV output feature. Replaced by JSON.
  • [ ] Remove HTML output feature. HTML plots will be generated by cargo-criterion.

Random wishlist

  • Unify Criterion and BenchmarkGroup such that there aren't two versions of sample_size, warm_up_time, etc.
  • Programmatic interface to benchmark results and baselines.
  • Better way of printing outliers, see #292.
  • Display both time and throughput, see #382.
  • Better alignment of output in the bencher format.
  • Consistent units for grouped benchmarks, see #367.

lemmih avatar Jul 27 '21 12:07 lemmih

My intention was actually to move away from relying on Gnuplot and make Plotters the default plotting backend; that way users don't need to install Gnuplot separately, but that's not set in stone. Is there some reason we want to avoid using Plotters?

bheisler avatar Jul 27 '21 21:07 bheisler

External installation of the gnuplot lib has always been a nuisance. Let's say that with plotters everything "just works".

c410-f3r avatar Aug 10 '21 15:08 c410-f3r

What's missing for a release of v0.4 ? I see all the TODOs are checked

poliorcetics avatar Jul 04 '22 07:07 poliorcetics

The code is there. Just needs to be reviewed.

lemmih avatar Jul 04 '22 07:07 lemmih

Can we use a JS library for plots so that they are more interactive?

https://github.com/bheisler/criterion.rs/issues/653

dclong avatar Feb 15 '23 08:02 dclong

I can understand CSV being an optional feature, but what is the reasoning behind it being removed? It is extremely useful for seamlessly importing and doing data visualization in other languages or programs (R, Python, Excel...).

wilgaboury avatar Sep 14 '23 23:09 wilgaboury