criterion.rs icon indicating copy to clipboard operation
criterion.rs copied to clipboard

Allow running benchmarks as tests under Miri

Open gefjon opened this issue 1 year ago • 0 comments

I have a set of benchmarks for some unsafe memory-management code. I also run these benches as tests via cargo test --benches. Ideally, I'd like to run the benchmarks as tests under Miri, but that doesn't seem to be possible with Criterion right now.

Specifically, criterion_plot::version unconditionally spawns a process to check for the Gnuplot version during Criterion::default, which leads through a long chain to an unsupported pipe2 call. Note that this happens even when passing --noplot or programmatically disabling plotting, as the only way to construct a Criterion appears to be by default.

The simplest way to avoid this, in my opinion, is likely to gate the current implementation of criterion_plot::version behind #[cfg(not(miri))], and to add a separate version behind #[cfg(miri)] which always returns an Err. I haven't tried this myself, so I don't know if that would be sufficient to run under Miri (given suitable command-line arguments), or if something else later down the line will cause issues.

I'd be happy to open a PR if that would help!

gefjon avatar May 02 '24 23:05 gefjon