BenchmarkTools.jl icon indicating copy to clipboard operation
BenchmarkTools.jl copied to clipboard

WIP: rebasing @localbtime and friends from LocalScopeBenchmarks.jl

Open miguelraz opened this issue 4 years ago • 2 comments

This is a WIP PR to merge a @localbtime and friends macro into BenchmarkTools.jl .

There's a thread with reported issues here and this PR took existing code from LocalScopeBenchmarks.jl, with all credit to @rdeits.

This PR:

  • adds a dependency on MacroTools and Ordered collections for the local sourcewalking helper functions
  • exports @localbenchmark, @localbtime, @localbelapsed
  • adds tests for said functions.
  • bumps the end position semver number by 1
  • changes the statistical metric for the local benchmarks to be the mean and not the median (seemed more robust locally, modulo CI woes.)

All feedback is welcome.

Another welcome feedback will be on how the new tests for local time regression measurement should be done: I found that when running them locally, the first run would pass all tests, then fail due only to the regression threshold set. I think it's hitting the microbenchmarks remembering cache history.

Lastly, this PR does not:

  • make a judgement on how to/if at all to deprecate @btime et al.

That is left as a discussion for another time, and for now I will focus on landing this functionality.

Ping also to @jlperla, @chrisrackauckas.

miguelraz avatar Oct 01 '19 17:10 miguelraz

The tests are failing / and passing due to a brittle judging function. One idea would be to judge loosely min, mean and median and if 2/3 pass then the judgement is considered invariant.

miguelraz avatar Oct 16 '19 03:10 miguelraz