is this correct samples with seconds behavior? - better document effect of tuning

[sbromberger]

function dosomething()
    sleep(1)
end

julia> @time @btime dosomething() samples=3 seconds=20
  1.002 s (7 allocations: 160 bytes)  # this is from btime
 26.740450 seconds (312.46 k allocations: 15.984 MiB, 1.59% gc time)  # this is from time

The documentation seems to imply that seconds is an upper bound on benchmarking, which leads me to believe that the proper behavior is to exit after n samples or m seconds, whichever comes first.

Am I misunderstanding how this works?

[vchuravy]

This is where @benchmark gives you a bit more info:

julia> function dosomething()
         count[] += 1
         sleep(1)
       end
dosomething (generic function with 1 method)

julia> count = Ref(0)
Base.RefValue{Int64}(0)

julia> @time @benchmark dosomething() samples=3 seconds=20
 26.778415 seconds (318.81 k allocations: 16.095 MiB, 1.74% gc time)
BenchmarkTools.Trial: 
  memory estimate:  128 bytes
  allocs estimate:  5
  --------------
  minimum time:     1.002 s (0.00% GC)
  median time:      1.002 s (0.00% GC)
  mean time:        1.002 s (0.00% GC)
  maximum time:     1.002 s (0.00% GC)
  --------------
  samples:          3
  evals/sample:     1

julia> count
Base.RefValue{Int64}(26)

So yes we did 3 samples and 1 evaluation per sample, but apparently we executed the code-snippet 26 times! This is the tuning phase of BenchmarkTools.

@btime and @benchmark can be expanded into:

julia> b = @benchmarkable dosomething() samples=3 seconds=20
Benchmark(evals=1, seconds=20.0, samples=3)

julia> tune!(b)
Benchmark(evals=1, seconds=20.0, samples=3)

julia> count[] = 0
0

julia> run(b)
BenchmarkTools.Trial: 
  memory estimate:  128 bytes
  allocs estimate:  5
  --------------
  minimum time:     1.002 s (0.00% GC)
  median time:      1.002 s (0.00% GC)
  mean time:        1.002 s (0.00% GC)
  maximum time:     1.002 s (0.00% GC)
  --------------
  samples:          3
  evals/sample:     1

julia> count
Base.RefValue{Int64}(3)

[sbromberger]

but apparently we executed the code-snippet 26 times! This is the tuning phase of BenchmarkTools.

I don't think that's right, unless the tuning phase is set to take up all the slack time (how would it do that?). If you set seconds to 60 instead of 20, for example, you run for 60 seconds.

[vchuravy]

(how would it do that?).

For a description of the tuning process see https://github.com/JuliaCI/BenchmarkTools.jl/blob/58af70428fbcbe4862453ddf135907ddc2032862/src/execution.jl#L151

or see the code at https://github.com/JuliaCI/BenchmarkTools.jl/blob/58af70428fbcbe4862453ddf135907ddc2032862/src/execution.jl#L135

The tuning phase ignores samples and uses the full seconds