PkgBenchmark.jl icon indicating copy to clipboard operation
PkgBenchmark.jl copied to clipboard

wrong sysimage for different julia command

Open hexaeder opened this issue 3 years ago • 1 comments

Hey! First of all, thanks for providing this nice package, it makes comparing of several commits really easy :)

I've tried to judge benchmarks for several julia versions. Unfortunately, this does not work due to

https://github.com/JuliaCI/PkgBenchmark.jl/blob/baa8bee9981ce46033d9a9427874d2eb24a207da/src/runbenchmark.jl#L249-L250

which takes the sysimage from the parent process.

julia> Base.julia_cmd(`julia-1.5`)
`julia-1.5 -Cnative -J/Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/sys.dylib -g1`

I'd argue, that a user supplied command should be taken as is without changes which depend on the parent command whatsoever. Maybe it would be better to default the juliacmd in BenchmarkConfig to Base.julia_cmd() and just do

 juliacmd = benchmarkconfig.juliacmd 

in the actual benchmark function?

hexaeder avatar Jul 05 '21 09:07 hexaeder

and just do

juliacmd = benchmarkconfig.juliacmd

I think that is how it used to work https://github.com/JuliaCI/PkgBenchmark.jl/commit/4113fe220383a7a9a6252161979a80adc80495e2#diff-21450190f4a3f8e55d48c877d0c9aba737cc12fb8c74afaa3cd28f3283669682L244 but it was changed. cc @omus.

KristofferC avatar Jul 05 '21 14:07 KristofferC