`tune!` ignores `evals=1` of `@benchmarkable`

[Seelengrab]

MWE:

julia> using BenchmarkTools

julia> function f(x)
          if !all(iszero, x)
             error("nonzero")
          else
             x[1] = 1
          end
       end
f (generic function with 1 method)

julia> bm = @benchmarkable f(x) setup=(x=zeros(3)) evals=1
Benchmark(evals=1, seconds=5.0, samples=10000)

julia> tune!(bm)
ERROR: nonzero
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:33
  [2] f
    @ ./REPL[6]:3 [inlined]
  [3] var"##core#282"(x::Vector{Float64})
    @ Main ~/.julia/packages/BenchmarkTools/uq9zP/src/execution.jl:479
  [4] var"##sample#283"(__params::BenchmarkTools.Parameters)
    @ Main ~/.julia/packages/BenchmarkTools/uq9zP/src/execution.jl:487
  [5] _lineartrial(b::BenchmarkTools.Benchmark, p::BenchmarkTools.Parameters; maxevals::Int64, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ BenchmarkTools ~/.julia/packages/BenchmarkTools/uq9zP/src/execution.jl:160
  [6] _lineartrial(b::BenchmarkTools.Benchmark, p::BenchmarkTools.Parameters)
    @ BenchmarkTools ~/.julia/packages/BenchmarkTools/uq9zP/src/execution.jl:152
  [7] #invokelatest#2
    @ ./essentials.jl:731 [inlined]
  [8] invokelatest
    @ ./essentials.jl:729 [inlined]
  [9] #lineartrial#46
    @ ~/.julia/packages/BenchmarkTools/uq9zP/src/execution.jl:34 [inlined]
 [10] lineartrial
    @ ~/.julia/packages/BenchmarkTools/uq9zP/src/execution.jl:34 [inlined]
 [11] tune!(b::BenchmarkTools.Benchmark, p::BenchmarkTools.Parameters; progressid::Nothing, nleaves::Float64, ndone::Float64, verbose::Bool, pad::String, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ BenchmarkTools ~/.julia/packages/BenchmarkTools/uq9zP/src/execution.jl:250
 [12] tune! (repeats 2 times)
    @ ~/.julia/packages/BenchmarkTools/uq9zP/src/execution.jl:249 [inlined]
 [13] top-level scope
    @ REPL[8]:1

[Seelengrab]

By printing x, I found out that this doesn't happen everytime..?

shell> cat bench.jl
using BenchmarkTools

function f(x)
  @show x
  if !all(iszero, x)
     error("nonzero")
  else
     x[1] = 1
  end
end

julia> tune!(bm)
x = [0.0]
x = [0.0]
x = [0.0]
x = [1.0]
ERROR: nonzero
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:33
  [2] f(x::Vector{Float64})
    @ Main ~/projects/Experiments/benchtune!/bench.jl:6
  [3] var"##core#282"(x::Vector{Float64})
    @ Main ~/.julia/dev/BenchmarkTools/src/execution.jl:480
  [4] var"##sample#283"(__params::BenchmarkTools.Parameters)
    @ Main ~/.julia/dev/BenchmarkTools/src/execution.jl:488
  [5] _lineartrial(b::BenchmarkTools.Benchmark, p::BenchmarkTools.Parameters; maxevals::Int64, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ BenchmarkTools ~/.julia/dev/BenchmarkTools/src/execution.jl:160
  [6] _lineartrial(b::BenchmarkTools.Benchmark, p::BenchmarkTools.Parameters)
    @ BenchmarkTools ~/.julia/dev/BenchmarkTools/src/execution.jl:152
  [7] #invokelatest#2
    @ ./essentials.jl:731 [inlined]
  [8] invokelatest
    @ ./essentials.jl:729 [inlined]
  [9] #lineartrial#46
    @ ~/.julia/dev/BenchmarkTools/src/execution.jl:34 [inlined]
 [10] lineartrial
    @ ~/.julia/dev/BenchmarkTools/src/execution.jl:34 [inlined]
 [11] tune!(b::BenchmarkTools.Benchmark, p::BenchmarkTools.Parameters; progressid::Nothing, nleaves::Float64, ndone::Float64, verbose::Bool, pad::String, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ BenchmarkTools ~/.julia/dev/BenchmarkTools/src/execution.jl:250
 [12] tune! (repeats 2 times)
    @ ~/.julia/dev/BenchmarkTools/src/execution.jl:249 [inlined]
 [13] top-level scope
    @ REPL[7]:1

This is super weird.

[Seelengrab]

Ok I think I found the culprit:

https://github.com/JuliaCI/BenchmarkTools.jl/blob/104f4c1e210da1933ace369d6db1393cf23ac102/src/execution.jl#L151-L165

That overwrites params.evals with the index of the array for some reason.

[Seelengrab]

Ok, I don't think I can just remove this. Subsequent iterations with more evals may be faster with more evals and there's no way to check if what we got passed in already had its evals value set since it seems to default to 1.

Welcome for any advice on how to deal with this - I'd like to use @benchmarkable together with evals=1 for benchmarking in my package and the function in question cannot handle more than one eval with the same input.

[vtjnash]

This could be documented clearer, with respect to mention the name of the parameter, as the point of tune is explicitly just to change this parameter to the best value for it https://juliaci.github.io/BenchmarkTools.jl/dev/reference/#BenchmarkTools.tune!-Tuple{BenchmarkGroup}

[Seelengrab]

I see, that's good to know. Is there a way tuning can be turned off in @benchmarkable when setting evals explicitly? As it is right now, running a @benchmarkable seems to require tuning it beforehand.