PkgBenchmark.jl
PkgBenchmark.jl copied to clipboard
wrong sysimage for different julia command
Hey! First of all, thanks for providing this nice package, it makes comparing of several commits really easy :)
I've tried to judge benchmarks for several julia versions. Unfortunately, this does not work due to
https://github.com/JuliaCI/PkgBenchmark.jl/blob/baa8bee9981ce46033d9a9427874d2eb24a207da/src/runbenchmark.jl#L249-L250
which takes the sysimage from the parent process.
julia> Base.julia_cmd(`julia-1.5`)
`julia-1.5 -Cnative -J/Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/sys.dylib -g1`
I'd argue, that a user supplied command should be taken as is without changes which depend on the parent command whatsoever.
Maybe it would be better to default the juliacmd
in BenchmarkConfig to Base.julia_cmd()
and just do
juliacmd = benchmarkconfig.juliacmd
in the actual benchmark function?
and just do
juliacmd = benchmarkconfig.juliacmd
I think that is how it used to work https://github.com/JuliaCI/PkgBenchmark.jl/commit/4113fe220383a7a9a6252161979a80adc80495e2#diff-21450190f4a3f8e55d48c877d0c9aba737cc12fb8c74afaa3cd28f3283669682L244 but it was changed. cc @omus.