Consider switching from Criterion to cargo bench

[jon-chuang]

Criterion sucks - it takes way too long to bench. I'm sure I've raised this issue before. It's very difficult to make settings that get around the whole annoying "statistical significance" mumbo jumbo it uses.

Google's Go bench framework is much better. I think I will finally get fed up enough that I'll rework all the benchmarks.

[ValarDragon]

The statistical significance measurements are very important for micro-benches though. Agreed that they don't give that much at the larger benchmark sizes, and just cause delays

[Pratyush]

One nice feature of criterion is that it's stateful: it automatically compares the differences between two benchmarks. With benchcmp I have to do that manually.

[jon-chuang]

@Pratyush I think this is a feature rather than a problem. Comparing manually just requires piping to a file. But criterion only allows for comparing two benchmarks. What if I want to compare 3 benchmarks, or a benchmark I had already run in the past? You can't do this with criterion.

[ValarDragon]

I have run into that problem as well, I'm not sure I'd classify the statefulness as a feature, since it does become a pain to track which change was benched last in order to get accurate timing measurements

[daira]

The statefulness is actually a pain. Typically, what you want to do is mark a particular revision and then compare the current state to that revision as you make changes. Criterion can't do that (AFAIK), which makes its comparison feature a bit of an "attractive nuisance" if you ask me. And yes, Criterion benchmarking does take ridiculously long (I know you can tweak iteration counts manually, but meh).