BenchmarkTools.jl icon indicating copy to clipboard operation
BenchmarkTools.jl copied to clipboard

Document necessity to interpolate variables into top-level execution environment

Open andreasnoack opened this issue 7 years ago • 2 comments

julia> [@elapsed n for n in 1:1]
1-element Array{Float64,1}:
 5.1e-8

julia> [@belapsed n for n in 1:1]
ERROR: UndefVarError: n not defined
Stacktrace:
 [1] ##core#667() at /Users/andreasnoack/.julia/v0.6/BenchmarkTools/src/execution.jl:316
 [2] ##sample#668(::BenchmarkTools.Parameters) at /Users/andreasnoack/.julia/v0.6/BenchmarkTools/src/execution.jl:322
 [3] #_run#10(::Bool, ::String, ::Array{Any,1}, ::Function, ::BenchmarkTools.Benchmark{Symbol("##benchmark#666")}, ::BenchmarkTools.Parameters) at /Users/andreasnoack/.julia/v0.6/BenchmarkTools/src/execution.jl:350
 [4] (::BenchmarkTools.#kw##_run)(::Array{Any,1}, ::BenchmarkTools.#_run, ::BenchmarkTools.Benchmark{Symbol("##benchmark#666")}, ::BenchmarkTools.Parameters) at ./<missing>:0
 [5] anonymous at ./<missing>:?
 [6] #run_result#19(::Array{Any,1}, ::Function, ::BenchmarkTools.Benchmark{Symbol("##benchmark#666")}, ::BenchmarkTools.Parameters) at /Users/andreasnoack/.julia/v0.6/BenchmarkTools/src/execution.jl:44
 [7] (::BenchmarkTools.#kw##run_result)(::Array{Any,1}, ::BenchmarkTools.#run_result, ::BenchmarkTools.Benchmark{Symbol("##benchmark#666")}, ::BenchmarkTools.Parameters) at ./<missing>:0
 [8] #run#21(::Array{Any,1}, ::Function, ::BenchmarkTools.Benchmark{Symbol("##benchmark#666")}, ::BenchmarkTools.Parameters) at /Users/andreasnoack/.julia/v0.6/BenchmarkTools/src/execution.jl:67
 [9] (::Base.#kw##run)(::Array{Any,1}, ::Base.#run, ::BenchmarkTools.Benchmark{Symbol("##benchmark#666")}, ::BenchmarkTools.Parameters) at ./<missing>:0 (repeats 2 times)
 [10] (::##8#9)(::Int64) at /Users/andreasnoack/.julia/v0.6/BenchmarkTools/src/execution.jl:234
 [11] collect(::Base.Generator{UnitRange{Int64},##8#9}) at ./array.jl:470

andreasnoack avatar Jun 26 '18 20:06 andreasnoack

Shouldn't that be done with interpolation?

julia> [@belapsed $n for n in 1:1]
1-element Array{Float64,1}:
 3.8e-11

maxbennedich avatar Sep 08 '18 09:09 maxbennedich

Benchmarking always occurs at top-level by design, this isn't a bug.

dup of https://github.com/JuliaCI/BenchmarkTools.jl/issues/109

see also https://github.com/JuliaCI/BenchmarkTools.jl/issues/65#issuecomment-313926098.

This really should be documented in more places (in the macro docstrings?) since people have run into it so many times.

jrevels avatar Sep 10 '18 15:09 jrevels