BenchmarkTools.jl icon indicating copy to clipboard operation
BenchmarkTools.jl copied to clipboard

Benchmarks only track minimum number of allocations and memory usage

Open Arkoniak opened this issue 3 years ago • 1 comments

It seems, that BenchmarkTools.Trial uses only single value for storing number of allocations and memory usage during trial runs. And judging from the results it track only minimum allocations and memory. Somewhat exaggerated example:

using BenchmarkTools

function f()
    n = rand() < 0.5 ? 1 : 100000
    return rand(n)
end

and result

julia> @benchmark f() evals = 1
BenchmarkTools.Trial:
  memory estimate:  64 bytes
  allocs estimate:  1
  --------------
  minimum time:     55.000 ns (0.00% GC)
  median time:      86.357 μs (0.00% GC)
  mean time:        56.463 μs (3.11% GC)
  maximum time:     573.892 μs (45.10% GC)
  --------------
  samples:          10000
  evals/sample:     1

which doesn't feel right.

Arkoniak avatar Feb 19 '22 18:02 Arkoniak