blake2_simd icon indicating copy to clipboard operation
blake2_simd copied to clipboard

migrate benchmarks to criterion

Open ordian opened this issue 4 years ago • 3 comments

https://github.com/bheisler/criterion.rs compiles on rust stable, meaning you can cargo check --benches in your CI to catch accidentally broken benches. You also get charts and statistics for free.

ordian avatar Sep 03 '19 21:09 ordian

I've used Criterion in a few places before. The reason I haven't used it here is that it seems to take a very long time to run by default, while the test::Bencher framework seems to give me numbers relatively quickly. At least in the context of this project, the results from test::Bencher seem to be pretty reliable, and it's been nice to be able to do really quick benchmarks while I play with inlining flags and things like that.

Maybe you can suggest some non-default parameters for Criterion that would make it run faster? But I don't know enough about the crate to know what to try.

oconnor663 avatar Sep 03 '19 23:09 oconnor663

I guess something like this should help:

criterion_group! {
	name = bench_multiprocess;
	config = dont_take_an_eternity_to_run();
	targets = blake2b_hash_many,
		hash_sneves_blake2sp,
		openssl_sha1,
}
criterion_main!(bench_multiprocess);

fn dont_take_an_eternity_to_run() -> Criterion {
	Criterion::default().nresamples(1_000)
		.without_plots()
		.sample_size(10)
}

For more options see https://bheisler.github.io/criterion.rs/criterion/struct.Criterion.html

ordian avatar Sep 04 '19 09:09 ordian

I'd be happy to take a pull request for this as long as we could keep the total benchmarking time under, say, 200% of what it is now.

oconnor663 avatar Sep 06 '19 18:09 oconnor663