JuliaPy
No way to install Julia signal handler when other dependency uses juliacall
Affects: JuliaCall
Describe the bug
Related to https://github.com/JuliaPy/PythonCall.jl/issues/219. There's currently a scenario with no possible workaround other than forcing a downstream user to set up environment variables to avoid a segfault.
Say that I import a package which uses serial Julia code. That package runs
from juliacall import Main as jl
to start Julia.
Then, I import a different package. That package calls multithreaded Julia code, and thus to avoid a segfault (https://juliapy.github.io/PythonCall.jl/stable/faq/#Is-PythonCall/JuliaCall-thread-safe?) instead performs the import instead as:
import os
os.environ["PYTHON_JULIACALL_HANDLE_SIGNALS"] = "yes"
from juliacall import Main as jl
However, at this point, the Julia runtime has already started. So now when I run a multithreaded portion of code, I will get a segfault, even though I implemented special workarounds in the multithreaded package.
How could I avoid this scenario? Indeed the user could always set up the environment variables themselves. But managing to figure this out from a random bus error will only be ~5% of users.
My temporary workaround is to put this code:
if "juliacall" in sys.modules:
warnings.warn(
"juliacall module already imported. Make sure that you have set `PYTHON_JULIACALL_HANDLE_SIGNALS=yes` to avoid segfaults."
)
before the import statement.
Well firstly, packages shouldn't generally be setting global config such as that env var, though I understand why you're doing it.
Over in #219 I suggested adding a warning to set PYTHON_JULIACALL_HANDLE_SIGNALS=yes if Julia is started with multiple threads. Do you think this would be a sufficient warning?
Also, given Julia starts with 1 thread by default, presumably you're already telling your users to set PYTHON_JULIACALL_NUM_THREADS? In which case you can also tell them to set PYHON_JULIACALL_HANDLE_SIGNALS?
If it helps, I was able to get a stacktrace for this:
Stacktrace
Stacktrace:
[1] wait
@ ./task.jl:352 [inlined]
[2] open_exclusive(path::String; mode::UInt16, poll_interval::Int64, wait::Bool, stale_age::Int64, refresh::Float64)
@ FileWatching.Pidfile ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:268
[3] open_exclusive
@ ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:232 [inlined]
[4] mkpidlock(at::String, pid::Int32; stale_age::Int64, refresh::Float64, kwopts::@Kwargs{})
@ FileWatching.Pidfile ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:67
[5] mkpidlock
@ ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:62 [inlined]
[6] mkpidlock(f::Pkg.Types.var"#51#54"{String, String, Dates.DateTime, String}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64})
@ FileWatching.Pidfile ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:91
[7] mkpidlock
@ ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:90 [inlined]
[8] mkpidlock
@ ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:88 [inlined]
[9] write_env_usage(source_file::String, usage_filepath::String)
@ Pkg.Types ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/Pkg/src/Types.jl:539
[10] Pkg.Types.EnvCache(env::Nothing)
@ Pkg.Types ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/Pkg/src/Types.jl:377
[11] EnvCache
@ ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/Pkg/src/Types.jl:356 [inlined]
[12] add_snapshot_to_undo(env::Nothing)
@ Pkg.API ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/Pkg/src/API.jl:2189
[13] add_snapshot_to_undo
@ ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/Pkg/src/API.jl:2185 [inlined]
[14] activate(path::String; shared::Bool, temp::Bool, io::Base.DevNull)
@ Pkg.API ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/Pkg/src/API.jl:1973
[15] top-level scope
@ none:7
nested task error: InterruptException:
Stacktrace:
[1] poptask(W::Base.IntrusiveLinkedListSynchronized{Task})
@ Base ./task.jl:985
[2] wait()
@ Base ./task.jl:994
[3] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool)
@ Base ./condition.jl:130
[4] wait
@ ./condition.jl:125 [inlined]
[5] _trywait(t::Timer)
@ Base ./asyncevent.jl:142
[6] wait
@ ./asyncevent.jl:159 [inlined]
[7] sleep(sec::Int64)
@ Base ./asyncevent.jl:265
[8] (::FileWatching.Pidfile.var"#13#14"{Int64, String})()
@ FileWatching.Pidfile ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:263
caused by: IOError: FileMonitor (start): no such file or directory (ENOENT)
Stacktrace:
[1] uv_error
@ ./libuv.jl:100 [inlined]
[2] start_watching(t::FileWatching.FileMonitor)
@ FileWatching ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/FileWatching.jl:525
[3] wait(m::FileWatching.FileMonitor)
@ FileWatching ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/FileWatching.jl:659
[4] watch_file(s::String, timeout_s::Float64)
@ FileWatching ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/FileWatching.jl:772
[5] watch_file
@ ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/FileWatching.jl:778 [inlined]
[6] (::FileWatching.Pidfile.var"#13#14"{Int64, String})()
@ FileWatching.Pidfile ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:260
This can also be avoided by setting the environment variable PYTHONFAULTHANDLER to some writable file. Credit to @willow-ahrens for some of the heavy lifting on this, cc @mtsokol.
