BenchmarkTools.jl
BenchmarkTools.jl copied to clipboard
Benchmarks only track minimum number of allocations and memory usage
It seems, that BenchmarkTools.Trial uses only single value for storing number of allocations and memory usage during trial runs. And judging from the results it track only minimum allocations and memory. Somewhat exaggerated example:
using BenchmarkTools
function f()
n = rand() < 0.5 ? 1 : 100000
return rand(n)
end
and result
julia> @benchmark f() evals = 1
BenchmarkTools.Trial:
memory estimate: 64 bytes
allocs estimate: 1
--------------
minimum time: 55.000 ns (0.00% GC)
median time: 86.357 μs (0.00% GC)
mean time: 56.463 μs (3.11% GC)
maximum time: 573.892 μs (45.10% GC)
--------------
samples: 10000
evals/sample: 1
which doesn't feel right.