benchee
benchee copied to clipboard
Feature suggestion: assert_all_same option to compare alternative implementations
Hi @PragTob, how are you doing?
What: an option like assert_all_same: true which would run all functions (for each input if using inputs) and ensure that results are equal for each input across functions.
Context: I've been mostly using Benchee to optimize pure functions in core, which means benchmarking alternative implementations. Failing early if one of my alternative implementation is not equivalent would improve the feedback loop, since I'm usually not writing tests yet during the benchmarking phase (but I often end up adding some dbg or manual checks).
I could write my own small wrapper, but since it might be a common use case I went for suggesting it in Benchee itself.
I'd be happy to send a PR if there is interest.