BenchmarkTools.jl
BenchmarkTools.jl copied to clipboard
WIP: rebasing @localbtime and friends from LocalScopeBenchmarks.jl
This is a WIP PR to merge a @localbtime
and friends macro into BenchmarkTools.jl .
There's a thread with reported issues here and this PR took existing code from LocalScopeBenchmarks.jl
, with all credit to @rdeits.
This PR:
- adds a dependency on MacroTools and Ordered collections for the local sourcewalking helper functions
- exports
@localbenchmark
,@localbtime
,@localbelapsed
- adds tests for said functions.
- bumps the end position semver number by
1
- changes the statistical metric for the local benchmarks to be the mean and not the median (seemed more robust locally, modulo CI woes.)
All feedback is welcome.
Another welcome feedback will be on how the new tests for local time regression measurement should be done: I found that when running them locally, the first run would pass all tests, then fail due only to the regression threshold set. I think it's hitting the microbenchmarks remembering cache history.
Lastly, this PR does not:
- make a judgement on how to/if at all to deprecate
@btime
et al.
That is left as a discussion for another time, and for now I will focus on landing this functionality.
Ping also to @jlperla, @chrisrackauckas.
The tests are failing / and passing due to a brittle judging function. One idea would be to judge loosely min, mean and median and if 2/3 pass then the judgement is considered invariant.