num-bigint icon indicating copy to clipboard operation
num-bigint copied to clipboard

use criterion.rs instead of the default bencher

Open lcnr opened this issue 5 years ago • 6 comments

The results of the default bencher are quite unstable(in successive benchmarks run without any other open programs, there were 10 % differences in the same test when comparing with benchcmp). In case you are interested, I would be glad to rewrite the current benches to use criterion and test if there are any improvement.

lcnr avatar Mar 26 '19 19:03 lcnr

I'm open to criterion. The only caveat is that dev-deps can't be optional, which means we'd only be able to test num-bigint according to their minimum version, currently rust 1.26. We can still do build-checks for older versions though, which is probably good enough.

Another option is to move the benchmarks to a sub-crate in the repo, which we'd never publish.

cuviper avatar Mar 28 '19 20:03 cuviper

I am somewhat in favor of using a sub crate, I will start implementing this in a private repo for now.

lcnr avatar Mar 29 '19 08:03 lcnr

I've converted the benchmarks to criterion: https://github.com/Lemmih/num-criterion. Any interest in moving this under the rust-num umbrella?

lemmih avatar Jul 18 '21 04:07 lemmih

@Lemmih I don't really want to get in the business of comparing with other implementations. Don't get me wrong, it's useful data to have, but I think that's better done on neutral ground.

cuviper avatar Jul 27 '21 21:07 cuviper

Without the shootout, would you be interested in moving it to rust-num?

lemmih avatar Jul 28 '21 01:07 lemmih

I'm open to a sub-crate in this repo as mentioned. I fear a separate repo will get neglected, and I'd rather have it live close to the code that it is meant to be testing.

cuviper avatar Aug 03 '21 21:08 cuviper