PackageCompiler spins infinitely when compiling sysimage

[eRedekopp]

I am trying to compile a sysimage using PackageCompiler but have been unable to successfully complete the process. Everything appears to work correctly until I reach "PackageCompiler: compiling incremental system image", at which point it spins forever. I left it overnight and had it running at least 11 hours before killing the process.

When I kill the process, it always gives me the following stacktrace, which appears to show the process stuck in a spinlock.

Stacktrace:
  [1] try_yieldto(undo::typeof(Base.ensure_rescheduled))
    @ Base ./task.jl:920
  [2] wait()
    @ Base ./task.jl:984
  [3] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool)
    @ Base ./condition.jl:130
  [4] wait
    @ ./condition.jl:125 [inlined]
  [5] wait(x::Base.Process)
    @ Base ./process.jl:661
  [6] success
    @ ./process.jl:523 [inlined]
  [7] run(::Cmd; wait::Bool)
    @ Base ./process.jl:480
  [8] run
    @ ./process.jl:477 [inlined]
  [9] #20
    @ ~/.julia/packages/PackageCompiler/b2smD/ext/TerminalSpinners.jl:157 [inlined]
 [10] spin(f::PackageCompiler.var"#20#22"{Cmd}, s::PackageCompiler.TerminalSpinners.Spinner{Base.TTY})
    @ PackageCompiler.TerminalSpinners ~/.julia/packages/PackageCompiler/b2smD/ext/TerminalSpinners.jl:164
 [11] macro expansion
    @ ~/.julia/packages/PackageCompiler/b2smD/ext/TerminalSpinners.jl:157 [inlined]
 [12] create_sysimg_object_file(object_file::String, packages::Vector{String}, packages_sysimg::Set{Base.PkgId}; project::String, base_sysimage::String, precompile_execution_file::Vector{String}, precompile_statements_file::Vector{String}, cpu_target::String, script::Nothing, sysimage_build_args::Cmd, extra_precompiles::String, incremental::Bool)
    @ PackageCompiler ~/.julia/packages/PackageCompiler/b2smD/src/PackageCompiler.jl:128
 [13] create_sysimg_object_file
    @ ~/.julia/packages/PackageCompiler/b2smD/src/PackageCompiler.jl:290 [inlined]
 [14] create_sysimage(packages::Vector{String}; sysimage_path::String, project::String, precompile_execution_file::Vector{String}, precompile_statements_file::Vector{String}, incremental::Bool, filter_stdlibs::Bool, cpu_target::String, script::Nothing, sysimage_build_args::Cmd, include_transitive_dependencies::Bool, base_sysimage::Nothing, julia_init_c_file::Nothing, version::Nothing, soname::Nothing, compat_level::String, extra_precompiles::String)
    @ PackageCompiler ~/.julia/packages/PackageCompiler/b2smD/src/PackageCompiler.jl:586
 [15] top-level scope
    @ REPL[8]:1

Julia Info

Julia Version 1.9.1 Commit 147bdf428cd (2023-06-07 08:27 UTC) Platform Info: OS: Linux (x86_64-linux-gnu) CPU: 2 × AMD EPYC 7B12 WORD_SIZE: 64 LIBM: libopenlibm LLVM: libLLVM-14.0.6 (ORCJIT, znver2) Threads: 1 on 2 virtual cores Environment: JULIA_IMAGE_THREADS = 1

PackageCompiler v2.1.7

Steps to reproduce

I have reproduced this error on multiple machines. The steps are as follows:

• Download a fresh install of Julia 1.9
• Open the REPL
• Open Pkg
• add StockFlow
• add PackageCompiler
• Close pkg
• using PackageCompiler
• create_sysimage(["StockFlow"]; sysimage_path="./image.so")

The infinite loop happens inside create_sysimage. This happens with both incremental and non-incremental builds.

[eRedekopp]

~~I have narrowed down the issue to StockFlow specifically. I am discussing with the package's maintainers. Any added information would be greatly appreciated~~

It seems to happen with other packages too. See below

[mcreel]

I also get a process that never seems to finish, when compiling the code at https://github.com/mcreel/Econometrics, using the MakeSysImage.jl script that is part of the code. With 1.8.5, it finishes in about 10 minutes.

[eRedekopp]

Thanks! I tried it with 1.6.0 and it compiled in a few minutes also. This seems to be an issue with version 2 specifically

[nsiccha]

I'm just here to write that I'm having the same problems (with Julia 1.9.0).

[rdiaz02]

I've had the same problem with even a simpler file, just following the example in https://julialang.github.io/PackageCompiler.jl/dev/examples/plots.html (I've documented it in https://discourse.julialang.org/t/sysimage-creation-example-with-plots-does-not-seem-to-work/101269).

But an even simpler example also seems to spin forever:

1. julia --history-file=no --startup-file=no

using PackageCompiler
create_sysimage(:Plots, sysimage_path="sys_plots.so")

Since this issue can be seen just with Plots, it is not necessarily due to StockFlow.

On the other hand, I can create sysimages for other packages; for example I can successfully reproduce the example in https://julialang.github.io/PackageCompiler.jl/dev/examples/ohmyrepl.html and I can also create sysimages that contain other packages, for example this works:

using PackageCompiler
create_sysimage([:LanguageServer, :SymbolServer];
                sysimage_path="sys_languagesymbolserver.so")

[eRedekopp]

Agreed -- I don't think this has to do with StockFlow specifically. I've edited my previous comment. I've played around with this quite a bit more and am having trouble finding patterns in when it does or does not work.

I ended up giving up on my original plans for this a couple weeks ago; instead I created a basic HTTP server that constantly runs and has everything pre-compiled, although this is not my preferred solution.

[rdiaz02]

In my case, removing all calls to Plots (or StatsPlots) seems to allow the creation of the sysimages I use. So that is what I am doing for now.

[dev10110]

@rdiaz02 yea that worked for me too. Not sure why.

[eRedekopp]

I'm not sure I understand that as a workaround. If we remove the calls to Plots from the sample script, doesn't that mean that the functions won't be pre-compiled, and so the only speedups will be in the 'using Plots' call and nothing else? In any case, I don't see how that would fix it for the minimal reproducible example in the original post

[rdiaz02]

@eRedekopp You are correct and this is not really a workaround, because of what you say. But, in the meantime, while this bug is fixed, removing Plots at least allows the creation of sysimages that use (at least some) other packages.

[eRedekopp]

Understood. Thanks for the clarification

[toollu]

Hitting the same after updating 1.8.5 to 1.9.2, also with Plots.jl in the Project. Another Project without Plots.jl works. Any ideas why @KristofferC?

[mcreel]

With Julia 1.10rc2, a project that includes Plots.jl successfully compiles faster than ever before! However, on the first try, it errors, and says to call Base.retry_load_extensions(). After doing this, on the second try, the compilation is successful. The specific script I used is https://github.com/mcreel/Econometrics/blob/main/MakeSysimage.jl, for a project with many dependencies, those in https://github.com/mcreel/Econometrics/blob/main/Project.toml

[BenCurran98]

I'm also having the issue with Plots.jl that @rdiaz02 described. Would be great to get a fix in Julia 1.9 out for this if possible? @KristofferC

[DilumAluthge]

It sounds like (for the Plots.jl problem), that Julia 1.9.0 is broken and Julia 1.10.0-rc2 is fixed?

If this is correct, then what would be really helpful is if someone could do a git bisect between Julia 1.9.0 and Julia 1.10.0-rc2 to figure out which PR fixed the issue. Then, the (usually) easy part is backporting that PR to Julia 1.9.x.

[michel2323]

I'm here to confirm this issue, too. The source code points to the C compiler call. The C compiler seems to not return. I tried different C compilers via JULIA_CC, but no changes.