BenchmarkTools.jl icon indicating copy to clipboard operation
BenchmarkTools.jl copied to clipboard

Repeated `@btime` with different setup behaves bad

Open sgaure opened this issue 2 months ago • 4 comments

function sumN(N)
    s = zero(N)
    for i in 1:N
        s += i
    end
    s
end

julia> @btime sumN(N) setup=(N=1_000_000)
  2.424 ns (0 allocations: 0 bytes)
500000500000

julia> @btime sumN(N) setup=(N=1_000_000e0)
  2.424 ns (0 allocations: 0 bytes)
500000500000

julia> @btime sumN(N) setup=(N=1_000_001e0)
  1.330 ms (0 allocations: 0 bytes)
5.00001500001e11

julia> @btime sumN(N) setup=(N=1_000_001)
  1.330 ms (0 allocations: 0 bytes)
5.00001500001e11

julia> @btime sumN(M) setup=(M=1_000_001)
  1.993 ns (0 allocations: 0 bytes)
500001500001

The Float64 trial should run slowly, and return a Float64, whereas the Int trial should run in a couple of ns and return an Int. Is there some caching of the setup going on, which uses == instead of ===?

sgaure avatar Oct 09 '25 07:10 sgaure