PythonCall.jl
PythonCall.jl copied to clipboard
JuliaPy
Segfault on propertynames
Affects: PythonCall / JuliaCall / Both [delete as appropriate]
Describe the bug
Using Python 3.12.8 and Julia 1.11, when I type <pyobject>. into the REPL, I see a segfault.
Your system
julia> versioninfo()
Julia Version 1.11.2
Commit 5e9a32e7af2 (2024-12-01 20:02 UTC)
Build Info:
Official https://julialang.org/ release
Platform Info:
OS: Linux (x86_64-linux-gnu)
CPU: 64 × AMD Ryzen Threadripper 3970X 32-Core Processor
WORD_SIZE: 64
LLVM: libLLVM-16.0.6 (ORCJIT, znver2)
Threads: 60 default, 0 interactive, 30 GC (on 64 virtual cores)
Environment:
JULIA_NUM_THREADS = 60
JULIA_PROJECT = @.
JULIA_EDITOR = emacs -nw
(...) pkg> st
Project HunFlair2 v0.1.0
Status `~/.../Project.toml`
[992eb4ea] CondaPkg v0.2.24
[6099a3de] PythonCall v0.9.23
(...) pkg> conda st
CondaPkg Status /.../CondaPkg.toml
Environment
/.../.CondaPkg/env
Pip packages
flair v0.14.0
Additional context
julia> s.
[57673] signal 11 (1): Segmentation fault
in expression starting at none:0
_PyInterpreterState_GET at /usr/local/src/conda/python-3.12.8/Include/internal/pycore_pystate.h:133 [inlined]
_PyType_Lookup at /usr/local/src/conda/python-3.12.8/Objects/typeobject.c:4729 [inlined]
_PyObject_LookupSpecial at /usr/local/src/conda/python-3.12.8/Objects/typeobject.c:2167
_dir_object at /usr/local/src/conda/python-3.12.8/Objects/object.c:1758 [inlined]
PyObject_Dir at /usr/local/src/conda/python-3.12.8/Objects/object.c:1790
PyObject_Dir at /home/tec/.julia/packages/PythonCall/Nr75f/src/C/pointers.jl:303 [inlined]
macro expansion at /home/tec/.julia/packages/PythonCall/Nr75f/src/Core/Py.jl:132 [inlined]
pydir at /home/tec/.julia/packages/PythonCall/Nr75f/src/Core/builtins.jl:216
propertynames at /home/tec/.julia/packages/PythonCall/Nr75f/src/Core/Py.jl:293
unknown function (ip: 0x7fab413505eb)
complete_symbol at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/REPL/src/REPLCompletions.jl:208
[...]
I see that crash trying to use the Matlab PythonAPI installed in a conda venv:
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.11.3 (2025-01-21)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
julia> using PythonCall
julia> me = pyimport("matlab.engine")
[long delay]
Python: <module 'matlab.engine' from '/home/sjbespa/.conda/envs/v3.12/lib/python3.12/site-packages/matlab/engine/__init__.py'>
julia> me.
[3507720] signal 11 (1): Segmentation fault
in expression starting at none:0
_PyInterpreterState_GET at /usr/local/src/conda/python-3.12.8/Include/internal/pycore_pystate.h:133 [inlined]
_PyType_Lookup at /usr/local/src/conda/python-3.12.8/Objects/typeobject.c:4729 [inlined]
_PyObject_LookupSpecial at /usr/local/src/conda/python-3.12.8/Objects/typeobject.c:2167
_dir_object at /usr/local/src/conda/python-3.12.8/Objects/object.c:1758 [inlined]
PyObject_Dir at /usr/local/src/conda/python-3.12.8/Objects/object.c:1790```
(@v1.11) pkg> status
Status ~/.julia/environments/v1.11/Project.toml
[8f4d0f93] Conda v1.10.2
[992eb4ea] CondaPkg v0.2.24
[497a8b3b] DoubleFloats v1.4.2
[6099a3de] PythonCall v0.9.24
Thanks both for reporting this. However I cannot reproduce this locally, e.g. I can do:
julia> using PythonCall
julia> x = pyint(3)
Python: 3
julia> x.
__abs__ __add__ __and__ __bool__ [etc.....]
@tecosaur You didn't specify what s is in your example. Does the error occur with any Python object, or just certain ones? I notice you started Julia with multithreading, do you still get the error with one thread?
@protogeezer I can't install matlabengine because I don't have matlab. Have you seen the problem elsewhere?
Edit: I tried reproducing on Windows and on Linux, with 1 thread and with many.
Fair enough. I submitted #591 (mentioned above) with more details. I tried the simple test you described - it didn't work here - it hung. However the regular expression example from the documentation still crashes as soon as I try to reference the imported regular expression module.
One additional detail is that I use python from Miniforge3. I do not use the base environment.
If you need further experiments, please let me know. Try more versions of Python? Do we need to use the "built in" version of python"?
I have contemplated getting more involved with this package because I depend on it so much. I would need some coaching to get off the ground. Actually even how to get on the runway... I did get to the point where I was fixing crashes in PyCall before PythonCall came along...
Stephen
(base) sjbespa@mbp-et ~ % julia
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.11.3 (2025-01-21)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
(@v1.11) pkg> build PythonCall
(@v1.11) pkg> build PythonPlot
julia> using PythonCall
julia> x = pyint(3)
Python: 3
julia> x.
^C
julia> re = pyimport("re")
Python: <module 're' from '/opt/Miniforge3/lib/python3.12/re/__init__.py'>
julia> re.
[6482] signal 11 (2): Segmentation fault: 11
in expression starting at none:0
_PyType_Lookup at /opt/Miniforge3/lib/libpython3.12.dylib (unknown line)
PyObject_Dir at /opt/Miniforge3/lib/libpython3.12.dylib (unknown line)
...
One additional detail is that I use python from Miniforge3.
Can you give the steps to reproduce your environment please? Including versions of downloaded software and any settings you set.
@tecosaur You didn't specify what s is in your example. Does the error occur with any Python object, or just certain ones? I notice you started Julia with multithreading, do you still get the error with one thread?
I had the impression it may be with any object, but it was originally with a module and I've just run into it again so I can give a more detailed reproducer:
~$ mkdir /tmp/pytest; cd /tmp/pytest
~$ julia +1.12 --startup-file=no --project=. -q
pkg> add PythonCall
julia> using PythonCall
pkg> conda pip_add DSPy
julia> const dspy = pyimport("dspy")
julia> dspy.<tab>
ERROR: Python: ImportError: C extension: pickle not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext' to build the C extensions first.
Python stacktrace: [...]
julia> exit()
[544212] signal 7 (128): Bus error
in expression starting at none:0
_PyErr_Occurred at /usr/local/src/conda/python-3.12.9/Include/internal/pycore_pyerrors.h:23 [inlined]
...
~$ julia +1.12 --startup-file=no --project=. -q
julia> using PythonCall
julia> const dspy = pyimport("dspy")
Python: <module 'dspy' ...>
julia> dspy.<tab>
[545473] signal 11 (1): Segmentation fault
in expression starting at none:0
_PyInterpreterState_GET at /usr/local/src/conda/python-3.12.9/Include/internal/pycore_pystate.h:133 [inlined]
_PyType_Lookup at /usr/local/src/conda/python-3.12.9/Objects/typeobject.c:4729 [inlined]
_PyObject_LookupSpecial at /usr/local/src/conda/python-3.12.9/Objects/typeobject.c:2167
_dir_object at /usr/local/src/conda/python-3.12.9/Objects/object.c:1758 [inlined]
PyObject_Dir at /usr/local/src/conda/python-3.12.9/Objects/object.c:1790
PyObject_Dir at /home/tec/.julia/packages/PythonCall/WMWY0/src/C/pointers.jl:303 [inlined]
macro expansion at /home/tec/.julia/packages/PythonCall/WMWY0/src/Core/Py.jl:132 [inlined]
pydir at /home/tec/.julia/packages/PythonCall/WMWY0/src/Core/builtins.jl:216
propertynames at /home/tec/.julia/packages/PythonCall/WMWY0/src/Core/Py.jl:293
Seems like this actually gives you two different errors for the price of one, fun! 😛
Since this was starting from an empty project using Julia 1.11.3 and the same system as the original issue, I think this should be all the relevant information?
I notice you started Julia with multithreading, do you still get the error with one thread?
Huh, it seems alright if only one thread is used.
Those examples are with Julia 1.12 which doesn't exist yet (??) do you get the same on 1.11?
Ah oops, I meant to use 1.11 but mis-typed. Yep, it's the same experience.
I have some news to report. Pursuant to Chris's request, a couple days ago, for the steps to reproduce my environment, what I was doing to create the recipe seems to have made things somewhat better.
Here is the example from the beginning of the PythonCall guide that didn't work (see the crash in a previous comment).
julia> using PythonCall
julia> re = pyimport("re")
Python: <module 're' from '/opt/Miniforge3/lib/python3.12/re/__init__.py'>
julia> words = re.findall("[a-zA-Z]+", "PythonCall.jl is very useful!")
Python: ['PythonCall', 'jl', 'is', 'very', 'useful']
julia> sentence = Py(" ").join(words)
Python: 'PythonCall jl is very useful'
julia> "PythonCall jl is very useful"
"PythonCall jl is very useful"
However, Chris's previous example still crashes:
julia> x = pyint(3)
Python: 3
julia> x.
[1338] signal 11 (2): Segmentation fault: 11
in expression starting at none:0
_PyType_Lookup at /opt/Miniforge3/lib/libpython3.12.dylib (unknown line)
PyObject_Dir at /opt/Miniforge3/lib/libpython3.12.dylib (unknown line)
PyObject_Dir at /Users/sjbespa/.julia/packages/PythonCall/WMWY0/src/C/pointers.jl:303 [inlined]
At this point I've tried deleting the "official" macOS julia app (which stopped working for some reason - the error message had something to do with not being able to launch an app(?) in a terminal window); several old versions of julia that I built for analysis (which has now been obviated by juliaup); and finally I deleted and rebuilt the 1.11 env and reinstalled PythonCall. I suppose I could try deleting .julia to see if the problem really is a software conflict rather than a bug.
I'll try to finish rooting out possible conflicts in the next few days.
Stephen
I have developed a simpler set of steps to cause the crash. This is what happens on the Dell XPS 17 3790 running Kubuntu 24.10:
(base) sjbespa@gauss:~$ rm -rf .julia
(base) sjbespa@gauss:~$ julia
Checking for new Julia versions
Installing Julia 1.11.3+0.x64.linux.gnu
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.11.3 (2025-01-21)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
(@v1.11) pkg> add PythonCall
Installing known registries into `~/.julia`
Added `General` registry to ~/.julia/registries
Updating registry at `~/.julia/registries/General.toml`
Resolving package versions...
...
julia> using PythonCall
julia> x = pyint(3)
Python: 3
julia> x.
[42889] signal 11 (1): Segmentation fault
in expression starting at none:0
_PyInterpreterState_GET at /usr/local/src/conda/python-3.12.9/Include/internal/pycore_pystate.h:133 [inlined]
_PyType_Lookup at /usr/local/src/conda/python-3.12.9/Objects/typeobject.c:4729 [inlined]
...
The paths in the call stack appear to be gibberish - the majority certainly aren't paths that exist on the machine.
Here are the relevant settings in the .bashrc file:
export JULIA_NUM_THREADS=$(nproc);
export PYTHONDIR="/opt/Miniforge3"
export PYTHONPATH="/opt/Miniforge3/bin/python3"
export MATLAB_HOME="/usr/local/MATLAB/2024b/"
export CONDA_JL_CONDA_EXE="/opt/Miniforge3/bin/conda"
#export CONDA_JL_USE_MINIFORGE=1
export JULIA_CONDAPKG_EXE="/opt/Miniforge3/bin/conda"
export JULIA_CONDAPKG_BACKEND="System"
export JULIA_PYTHONCALL_EXE=$PYTHONPATH
export PATH="$PATH:$MATLAB_HOME/bin:"
export PATH="/opt/Miniforge3/bin:/opt/Miniforge3/sbin:$PATH"
export PATH="/opt/Miniforge3/lib:/opt/Miniforge3/libexec:$PATH"
These variable setting appear to work on macOS Sequoia (15.3) on a MBP with Apple Silicon. I'll continue to try to narrow down the example. Both Miniforge3 is writeable on both machines.
I hope this is useful.
Stephen
This somehow could prevent the Segmentation fault
struct Module
name::Symbol
py::Py
attributes::Ref{Vector{Symbol}}
end
Module(name) = Module(name, pynew(), Ref{Vector{Symbol}}())
const numpy = Module(:numpy)
is_public_attribute(name) = !startswith(string(name), "__")
function __init__()
PythonCall.pycopy!(numpy.py, pyimport("numpy"))
numpy.attributes[] = filter(is_public_attribute, propertynames(numpy.py))
end
Base.propertynames(m::Module) = m.attributes[]
@Beforerr: could you be a bit more specific about where your code should go?
With more testing I could soon conclude that the crash tecosaur and I reported depends only on the version of python that PythonCall is configured with. So far, I have been unable to reproduce the crash with python 3.11. Not so with Python 3.12. So far this pattern holds on a MacBook Pro M1 running sequoia, and a Dell with an i9 running Ubuntu 24.10.
Stephen
It is just a workaround (putting attributes into Julia so propertynames would not call py related function).
P.S:. It also happens in Mac Studio M2, M1 and Python 3.11
@beforerr: That's disappointing. If it's not the version of python, it could mean it's the values in the environment variables for Conda, CondaPkg and/or PythonCall? I'll experiment more with that in mind. Just now I was able to do the example of assigning a pyint to x, then hitting x.
If you add a comment with the steps to lead to the crashes that you are seeing, I'll try them too, and the workaround you proposed.
Stephen
@cjdoris: I propose that we use the debugger at this point. I don't know how to do that - so far VSCode has done nicely for my purposes. If you explain how this could be done, I'd certainly work to better document the cause of the crashes.
Stephen
I suspect that this is related to Julia / Python OpenSSL compatibility. On Windows with a Pixi install I see this if I conda add python=3.12, but it disappears if I go down to conda add python=3.12.0. This is the only Python 3.12 that will go down to OpenSSL v3.1.
I don't quite understand how this works but I thought CondaPkg would restrict OpenSSL more with https://github.com/JuliaPy/CondaPkg.jl/pull/140.
@visr that's interesting - can you show the logs showing CondaPkg installing an environment where this fails? i.e. with python=3.12
Hmm for a reason I don't understand 1.12 adds openssl = ">=3, <3.1" and 1.11 doesn't. Is that because OpenSSL_jll is only in my Manifest on 1.12? In any case to be able to test things out I set JULIA_CONDAPKG_OPENSSL_VERSION = ignore.
Interestingly the same segfault occurs on 1.12, and using OpenSSL before using PythonCall only seems to stop the error from happening on 1.11, not on 1.12.
Here is a session on 1.11 where I get Python 3.12.10 despite not setting JULIA_CONDAPKG_OPENSSL_VERSION, which also shows the segfault. Is that the log you are interested in?
Details
❯ julia +1.11
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.11.4 (2025-03-10)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
julia> versioninfo()
Julia Version 1.11.4
Commit 8561cc3d68 (2025-03-10 11:36 UTC)
Build Info:
Official https://julialang.org/ release
Platform Info:
OS: Windows (x86_64-w64-mingw32)
CPU: 24 × 12th Gen Intel(R) Core(TM) i7-12800HX
WORD_SIZE: 64
LLVM: libLLVM-16.0.6 (ORCJIT, alderlake)
Threads: 12 default, 0 interactive, 6 GC (on 24 virtual cores)
Environment:
JULIA_DEPOT_PATH = C:\ProgramData\DevDrives\.julia;
JULIA_EDITOR = code
JULIA_NUM_THREADS = 12
JULIA_PKG_PRESERVE_TIERED_INSTALLED = true
JULIA_PKG_USE_CLI_GIT = true
JULIA_PROJECT = @.
(@v1.11) pkg> activate .
Activating new project at `C:\ProgramData\DevDrives\temp\2`
(2) pkg> add CondaPkg PythonCall
Resolving package versions...
Updating `C:\ProgramData\DevDrives\temp\2\Project.toml`
[992eb4ea] + CondaPkg v0.2.28
[6099a3de] + PythonCall v0.9.24
Updating `C:\ProgramData\DevDrives\temp\2\Manifest.toml`
[992eb4ea] + CondaPkg v0.2.28
[9a962f9c] + DataAPI v1.16.0
[e2d170a0] + DataValueInterfaces v1.0.0
[82899510] + IteratorInterfaceExtensions v1.0.0
[692b3bcd] + JLLWrappers v1.7.0
[0f8b85d8] + JSON3 v1.14.2
[1914dd2f] + MacroTools v0.5.15
[0b3b1443] + MicroMamba v0.1.14
[bac558e1] + OrderedCollections v1.8.0
[69de0a69] + Parsers v2.8.1
[fa939f87] + Pidfile v1.3.0
⌅ [aea7be01] + PrecompileTools v1.2.1
[21216c6a] + Preferences v1.4.3
[6099a3de] + PythonCall v0.9.24
[ae029012] + Requires v1.3.1
[6c6a2e73] + Scratch v1.2.1
[856f2bd8] + StructTypes v1.11.0
[3783bdb8] + TableTraits v1.0.1
[bd369af6] + Tables v1.12.0
[e17b2a0c] + UnsafePointers v1.0.0
[f8abcde7] + micromamba_jll v1.5.8+0
[4d7b5844] + pixi_jll v0.41.3+0
[0dad84c5] + ArgTools v1.1.2
[56f22d72] + Artifacts v1.11.0
[2a0f44e3] + Base64 v1.11.0
[ade2ca70] + Dates v1.11.0
[f43a241f] + Downloads v1.6.0
[7b1f6079] + FileWatching v1.11.0
[b77e0a4c] + InteractiveUtils v1.11.0
[4af54fe1] + LazyArtifacts v1.11.0
[b27032c2] + LibCURL v0.6.4
[76f85450] + LibGit2 v1.11.0
[8f399da3] + Libdl v1.11.0
[56ddb016] + Logging v1.11.0
[d6f4376e] + Markdown v1.11.0
[a63ad114] + Mmap v1.11.0
[ca575930] + NetworkOptions v1.2.0
[44cfe95a] + Pkg v1.11.0
[de0858da] + Printf v1.11.0
[9a3f8284] + Random v1.11.0
[ea8e919c] + SHA v0.7.0
[9e88b42a] + Serialization v1.11.0
[fa267f1f] + TOML v1.0.3
[a4e569a6] + Tar v1.10.0
[8dfed614] + Test v1.11.0
[cf7118a7] + UUIDs v1.11.0
[4ec0a83e] + Unicode v1.11.0
[deac9b47] + LibCURL_jll v8.6.0+0
[e37daf67] + LibGit2_jll v1.7.2+0
[29816b5a] + LibSSH2_jll v1.11.0+1
[c8ffd9c3] + MbedTLS_jll v2.28.6+0
[14a3606d] + MozillaCACerts_jll v2023.12.12
[83775a58] + Zlib_jll v1.2.13+1
[8e850ede] + nghttp2_jll v1.59.0+0
[3f19e933] + p7zip_jll v17.4.0+2
Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading. To see why use `status --outdated -m`
julia> using CondaPkg
(2) pkg> conda add python
CondaPkg Found dependencies: C:\ProgramData\DevDrives\temp\2\CondaPkg.toml
CondaPkg Found dependencies: C:\ProgramData\DevDrives\.julia\packages\PythonCall\WMWY0\CondaPkg.toml
CondaPkg Resolving changes
+ python
CondaPkg Initialising pixi
│ C:\ProgramData\DevDrives\.julia\artifacts\aa881b8313f97d7b8797934574a1cfe551f85222\bin\pixi.exe
│ init
│ --format pixi
└ C:\ProgramData\DevDrives\temp\2\.CondaPkg
✔ Created C:\ProgramData\DevDrives\temp\2\.CondaPkg\pixi.toml
CondaPkg Wrote C:\ProgramData\DevDrives\temp\2\.CondaPkg\pixi.toml
│
│ [dependencies.python]
│ channel = "conda-forge"
│ build = "*cpython*"
│ version = ">=3.8,<4"
│ [project]
│ name = ".CondaPkg"
│ platforms = ["win-64"]
│ channels = ["conda-forge"]
│ channel-priority = "strict"
└ description = "automatically generated by CondaPkg.jl"
CondaPkg Installing packages
│ C:\ProgramData\DevDrives\.julia\artifacts\aa881b8313f97d7b8797934574a1cfe551f85222\bin\pixi.exe
│ install
└ --manifest-path C:\ProgramData\DevDrives\temp\2\.CondaPkg\pixi.toml
✔ The default environment has been installed.
(2) pkg> conda st
CondaPkg Status C:\ProgramData\DevDrives\temp\2\CondaPkg.toml
Environment
C:\ProgramData\DevDrives\temp\2\.CondaPkg\.pixi\envs\default
Packages
python v3.12.10
julia> using PythonCall
julia> x=pyint(3)
Python: 3
julia> x.
Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x7fff977c6cbd -- PyType_Lookup at C:\ProgramData\DevDrives\temp\2\.CondaPkg\.pixi\envs\default\python312.dll (unknown line)
in expression starting at none:0
PyType_Lookup at C:\ProgramData\DevDrives\temp\2\.CondaPkg\.pixi\envs\default\python312.dll (unknown line)
PyObject_Dir at C:\ProgramData\DevDrives\temp\2\.CondaPkg\.pixi\envs\default\python312.dll (unknown line)
PyObject_Dir at C:\ProgramData\DevDrives\.julia\packages\PythonCall\WMWY0\src\C\pointers.jl:303 [inlined]
macro expansion at C:\ProgramData\DevDrives\.julia\packages\PythonCall\WMWY0\src\Core\Py.jl:132 [inlined]
pydir at C:\ProgramData\DevDrives\.julia\packages\PythonCall\WMWY0\src\Core\builtins.jl:216
propertynames at C:\ProgramData\DevDrives\.julia\packages\PythonCall\WMWY0\src\Core\Py.jl:293
unknown function (ip: 000001467ac858b2)
complete_symbol at C:\workdir\usr\share\julia\stdlib\v1.11\REPL\src\REPLCompletions.jl:208
#complete_identifiers!#57 at C:\workdir\usr\share\julia\stdlib\v1.11\REPL\src\REPLCompletions.jl:1182
complete_identifiers! at C:\workdir\usr\share\julia\stdlib\v1.11\REPL\src\REPLCompletions.jl:1082 [inlined]
completions at C:\workdir\usr\share\julia\stdlib\v1.11\REPL\src\REPLCompletions.jl:1439
#complete_line#85 at C:\workdir\usr\share\julia\stdlib\v1.11\REPL\src\REPL.jl:637
complete_line at C:\workdir\usr\share\julia\stdlib\v1.11\REPL\src\REPL.jl:634
jfptr_complete_line_10909.1 at C:\ProgramData\DevDrives\.julia\juliaup\julia-1.11.4+0.x64.w64.mingw32\share\julia\compiled\v1.11\REPL\u0gqU_hz07T.dll (unknown line)
check_for_hint at C:\workdir\usr\share\julia\stdlib\v1.11\REPL\src\LineEdit.jl:387
#143 at C:\workdir\usr\share\julia\stdlib\v1.11\REPL\src\LineEdit.jl:2527
jl_apply at C:/workdir/src\julia.h:2157 [inlined]
jl_f__call_latest at C:/workdir/src\builtins.c:875
#invokelatest#2 at .\essentials.jl:1055 [inlined]
invokelatest at .\essentials.jl:1052 [inlined]
#30 at C:\workdir\usr\share\julia\stdlib\v1.11\REPL\src\LineEdit.jl:1711
jfptr_YY.30_8434.1 at C:\ProgramData\DevDrives\.julia\juliaup\julia-1.11.4+0.x64.w64.mingw32\share\julia\compiled\v1.11\REPL\u0gqU_hz07T.dll (unknown line)
macro expansion at C:\workdir\usr\share\julia\stdlib\v1.11\REPL\src\LineEdit.jl:2861 [inlined]
macro expansion at .\lock.jl:273 [inlined]
#282 at C:\workdir\usr\share\julia\stdlib\v1.11\REPL\src\LineEdit.jl:2851
jfptr_YY.282_9009.1 at C:\ProgramData\DevDrives\.julia\juliaup\julia-1.11.4+0.x64.w64.mingw32\share\julia\compiled\v1.11\REPL\u0gqU_hz07T.dll (unknown line)
jl_apply at C:/workdir/src\julia.h:2157 [inlined]
start_task at C:/workdir/src\task.c:1202
Allocations: 13938366 (Pool: 13936399; Big: 1967); GC: 21
Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x7fff978e27c4 -- Py_FinalizeEx at C:\ProgramData\DevDrives\temp\2\.CondaPkg\.pixi\envs\default\python312.dll (unknown line)
in expression starting at none:0
Py_FinalizeEx at C:\ProgramData\DevDrives\temp\2\.CondaPkg\.pixi\envs\default\python312.dll (unknown line)
Py_FinalizeEx at C:\ProgramData\DevDrives\.julia\packages\PythonCall\WMWY0\src\C\pointers.jl:303 [inlined]
#37 at C:\ProgramData\DevDrives\.julia\packages\PythonCall\WMWY0\src\C\context.jl:194
unknown function (ip: 000001467ac85e64)
_atexit at .\initdefs.jl:459
jfptr__atexit_71343.1 at C:\ProgramData\DevDrives\.julia\juliaup\julia-1.11.4+0.x64.w64.mingw32\lib\julia\sys.dll (unknown line)
jl_apply at C:/workdir/src\julia.h:2157 [inlined]
ijl_atexit_hook at C:/workdir/src\init.c:271
ijl_exit at C:/workdir/src\init.c:207
jl_exception_handler at C:/workdir/src\signals-win.c:337 [inlined]
jl_exception_handler at C:/workdir/src\signals-win.c:229
__julia_personality at C:/workdir/src\win32_ucontext.c:28
_chkstk at C:\WINDOWS\SYSTEM32\ntdll.dll (unknown line)
RtlWow64GetCurrentCpuArea at C:\WINDOWS\SYSTEM32\ntdll.dll (unknown line)
KiUserExceptionDispatcher at C:\WINDOWS\SYSTEM32\ntdll.dll (unknown line)
PyType_Lookup at C:\ProgramData\DevDrives\temp\2\.CondaPkg\.pixi\envs\default\python312.dll (unknown line)
PyObject_Dir at C:\ProgramData\DevDrives\temp\2\.CondaPkg\.pixi\envs\default\python312.dll (unknown line)
PyObject_Dir at C:\ProgramData\DevDrives\.julia\packages\PythonCall\WMWY0\src\C\pointers.jl:303 [inlined]
macro expansion at C:\ProgramData\DevDrives\.julia\packages\PythonCall\WMWY0\src\Core\Py.jl:132 [inlined]
pydir at C:\ProgramData\DevDrives\.julia\packages\PythonCall\WMWY0\src\Core\builtins.jl:216
propertynames at C:\ProgramData\DevDrives\.julia\packages\PythonCall\WMWY0\src\Core\Py.jl:293
unknown function (ip: 000001467ac858b2)
complete_symbol at C:\workdir\usr\share\julia\stdlib\v1.11\REPL\src\REPLCompletions.jl:208
#complete_identifiers!#57 at C:\workdir\usr\share\julia\stdlib\v1.11\REPL\src\REPLCompletions.jl:1182
complete_identifiers! at C:\workdir\usr\share\julia\stdlib\v1.11\REPL\src\REPLCompletions.jl:1082 [inlined]
completions at C:\workdir\usr\share\julia\stdlib\v1.11\REPL\src\REPLCompletions.jl:1439
#complete_line#85 at C:\workdir\usr\share\julia\stdlib\v1.11\REPL\src\REPL.jl:637
complete_line at C:\workdir\usr\share\julia\stdlib\v1.11\REPL\src\REPL.jl:634
jfptr_complete_line_10909.1 at C:\ProgramData\DevDrives\.julia\juliaup\julia-1.11.4+0.x64.w64.mingw32\share\julia\compiled\v1.11\REPL\u0gqU_hz07T.dll (unknown line)
check_for_hint at C:\workdir\usr\share\julia\stdlib\v1.11\REPL\src\LineEdit.jl:387
#143 at C:\workdir\usr\share\julia\stdlib\v1.11\REPL\src\LineEdit.jl:2527
jl_apply at C:/workdir/src\julia.h:2157 [inlined]
jl_f__call_latest at C:/workdir/src\builtins.c:875
#invokelatest#2 at .\essentials.jl:1055 [inlined]
invokelatest at .\essentials.jl:1052 [inlined]
#30 at C:\workdir\usr\share\julia\stdlib\v1.11\REPL\src\LineEdit.jl:1711
jfptr_YY.30_8434.1 at C:\ProgramData\DevDrives\.julia\juliaup\julia-1.11.4+0.x64.w64.mingw32\share\julia\compiled\v1.11\REPL\u0gqU_hz07T.dll (unknown line)
macro expansion at C:\workdir\usr\share\julia\stdlib\v1.11\REPL\src\LineEdit.jl:2861 [inlined]
macro expansion at .\lock.jl:273 [inlined]
#282 at C:\workdir\usr\share\julia\stdlib\v1.11\REPL\src\LineEdit.jl:2851
jfptr_YY.282_9009.1 at C:\ProgramData\DevDrives\.julia\juliaup\julia-1.11.4+0.x64.w64.mingw32\share\julia\compiled\v1.11\REPL\u0gqU_hz07T.dll (unknown line)
jl_apply at C:/workdir/src\julia.h:2157 [inlined]
start_task at C:/workdir/src\task.c:1202
Allocations: 13938366 (Pool: 13936399; Big: 1967); GC: 21
@visr thanks for that, but I cannot reproduce on my Windows laptop ☹. I followed the exact same steps and it works fine for me.
julia> versioninfo()
Julia Version 1.11.4
Commit 8561cc3d68 (2025-03-10 11:36 UTC)
Build Info:
Official https://julialang.org/ release
Platform Info:
OS: Windows (x86_64-w64-mingw32)
CPU: 12 × Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz
WORD_SIZE: 64
LLVM: libLLVM-16.0.6 (ORCJIT, skylake)
Threads: 12 default, 1 interactive, 6 GC (on 12 virtual cores)
Thanks for trying and including versioninfo. I noticed you run with 1 interactive thread and I with 0. If I add an interactive thread, the segfault goes away.
This may explain you seeing it with different thread numbers in https://github.com/JuliaPy/PythonCall.jl/issues/586#issuecomment-2676121449 and others seeing it disappear with one thread in https://github.com/JuliaPy/PythonCall.jl/issues/586#issuecomment-2677498064 (n interactive threads was unspecified).
Running some different combinations I get
works
--threads=1
--threads=2
--threads=3
--threads=3,1
--threads=4,1
--threads=12,1
segfault
--threads=4
--threads=1,2
--threads=3,2
seen both
--threads=5
So there may be a bit of noise there but it is quite consistent: 0 interactive threads: works if 4 or more in default pool? 1 interactive thread: works 2 interactive threads: segfaults
I came into this thread suggesting it is an OpenSSL thing in https://github.com/JuliaPy/PythonCall.jl/issues/586#issuecomment-2715781804, and was hoping to try this again with OpenSSL.jl v3.5. But I cannot reproduce that anymore on latest Julia/PythonCall versions, I am seeing segfaults with old and new OpenSSL versions in my Conda environment.
So I worked around this now by changing JULIA_NUM_THREADS=12 to JULIA_NUM_THREADS=12,1. I have no idea how this could trigger this issue though.
I wonder if propertynames is being called on a different thread to the thread where Python was started.
Can you try adding @show threadid() to https://github.com/JuliaPy/PythonCall.jl/blob/c84c05954bf5e8da93e2e8efacd4177c032b0465/src/Core/Py.jl#L287 and https://github.com/JuliaPy/PythonCall.jl/blob/c84c05954bf5e8da93e2e8efacd4177c032b0465/src/C/context.jl#L33 to test this? Compare the results between the cases you tried above. And also report whether the reported thread is interactive or not.
Yeah that seems to be spot on. Ran it with c934d808e1ecb37d9c3d11f50b9939c4d8e9d32a:
Works with 1 interactive thread when both run on the same inteactive thread:
julia> using PythonCall
┌ Info: init_context()
│ threadid() = 1
│ nthreads(:interactive) = 1
└ nthreads(:default) = 12
julia> x = pyint(3)
Python: 3
julia> x.┌ Info: propertynames(x::Py)
│ threadid() = 1
│ nthreads(:interactive) = 1
└ nthreads(:default) = 12
__abs__ __add__
Fails with 0 or more than 1 interactive threads when both run on different threads:
julia> using PythonCall
┌ Info: init_context()
│ threadid() = 1
│ nthreads(:interactive) = 2
└ nthreads(:default) = 12
julia> x=pyint(3)
Python: 3
julia> x.┌ Info: propertynames(x::Py)
│ threadid() = 2
│ nthreads(:interactive) = 2
└ nthreads(:default) = 12
Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION
Thanks, I can reproduce this on Linux now.
OK so we can conclude that if there are multiple interactive threads then tab-completion and package-importing can happen on different threads, which is bad. And if there are none, then presumably Julia puts interactive stuff on the default threads, so if there are multiple of those then same problem. So we require exactly one interactive thread, or exactly one thread.
I don't think there's anything that we can fix in PythonCall is there? I think all we can do is detect that this is occurring (by saving threadid() at init and checking if it's different in propertynames(::Py)) and print a warning with some advice. And put this info into the FAQ.
OK maybe there is something we can do for this specific issue - have a long-running task whose job is to compute propertynames(x) where x is read from a channel, writing the result to another channel. Then inside propertynames(::Py), check if we are on the "main" thread, if so continue as normal, but if not then fire a query off to this task.
Calling propertynames(x) doesn't segfault, but via REPLCompletions it does. See the full stacktrace for the REPLCompletions path: https://github.com/JuliaPy/PythonCall.jl/issues/586#issuecomment-2799027653
I don't know much about the REPL or how thread safety is supported in this package besides reading the multi-threading docs. But do I correctly understand that calling Python concurrently is only safe if the user locks the GIL? So if say the REPL does concurrent Python calls via e.g. propertynames, it cannot know about this and thus triggers the segfault?
I see #627 has some discussion on using Julia locks on the GIL, I supposed that may help here as well?
There is also https://github.com/JuliaLang/julia/issues/50084 which suggests defaulting to 1 interactive thread. Won't help with 2+ interactive threads specified by the user, but I assume most run into this because they use JULIA_NUM_THREADS=x. Luckily from Julia 1.12 JULIA_NUM_THREADS=x will default to 1 interactive thread.
After reading https://github.com/JuliaPy/PythonCall.jl/pull/635#issuecomment-3042334536 I looked a bit more into the GIL with these Python docs.
The tab completion works if I we can acquire the GIL in the non-main Julia thread that calls propertynames. We cannot acquire it since the main Julia thread locks it by default. But if I manually release it from the main thread, and wrap the body of propertynames in GIL.@lock it works.
using PythonCall
using PythonCall: GIL, C
x = pyint(3)
# unlock the GIL
state = C.PyEval_SaveThread()
x.<tab> # works!
Of course I'd have to reacquire the GIL if you want to run any other Python code from Julia. But it makes me wonder if we should release the GIL by default, and lock if for any Python C API call we do.
There is also JuliaLang/julia#50084 which suggests defaulting to 1 interactive thread. Won't help with 2+ interactive threads specified by the user, but I assume most run into this because they use
JULIA_NUM_THREADS=x. Luckily from Julia 1.12JULIA_NUM_THREADS=xwill default to 1 interactive thread.
I'm running Julia 1.12-rc3, and for me the default setting of 1 interactive thread doesn't work 🤔 Unless I start Julia with -t 1 it'll still segfault on tab-completion.
Same here on Macos, julia -tauto,1, verified that there's only one interactive thread, still segfaults when doing
using PythonCall
pd = pyimport("pandas")
pd.
results in
[84961] signal 11 (2): Segmentation fault: 11
in expression starting at REPL[7]:1
_PyType_LookupRef at /Users/<user>/.julia/dev/Dev/.CondaPkg/.pixi/envs/default/lib/libpython3.13.dylib (unknown line)
PyObject_Dir at /Users/<user>/.julia/dev/Dev/.CondaPkg/.pixi/envs/default/lib/libpython3.13.dylib (unknown line)
PyObject_Dir at /Users/<user>/.julia/packages/PythonCall/mkWc2/src/C/pointers.jl:300 [inlined]
macro expansion at /Users/<user>/.julia/packages/PythonCall/mkWc2/src/Core/Py.jl:118 [inlined]
pydir at /Users/<user>/.julia/packages/PythonCall/mkWc2/src/Core/builtins.jl:191
propertynames at /Users/<user>/.julia/packages/PythonCall/mkWc2/src/Core/Py.jl:270
unknown function (ip: 0x132da034f) at (unknown file)
#complete_symbol!#6 at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-12/usr/share/julia/stdlib/v1.12/REPL/src/REPLCompletions.jl:230
complete_symbol! at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-12/usr/share/julia/stdlib/v1.12/REPL/src/REPLCompletions.jl:184
unknown function (ip: 0x132d8807f) at (unknown file)
completions at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-12/usr/share/julia/stdlib/v1.12/REPL/src/REPLCompletions.jl:1150
#complete_line#66 at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-12/usr/share/julia/stdlib/v1.12/REPL/src/REPL.jl:809
complete_line at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-12/usr/share/julia/stdlib/v1.12/REPL/src/REPL.jl:807
unknown function (ip: 0x132b48533) at (unknown file)
#complete_line_named#17 at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-12/usr/share/julia/stdlib/v1.12/REPL/src/LineEdit.jl:401
complete_line_named at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-12/usr/share/julia/stdlib/v1.12/REPL/src/LineEdit.jl:400 [inlined]
macro expansion at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-12/usr/share/julia/stdlib/v1.12/REPL/src/LineEdit.jl:442 [inlined]
macro expansion at ./lock.jl:376 [inlined]
#check_show_hint##2 at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-12/usr/share/julia/stdlib/v1.12/REPL/src/LineEdit.jl:439
jfptr_YY.check_show_hintYY.YY.2_18133.1 at /Users/<user>/.julia/juliaup/julia-1.12.1+0.aarch64.apple.darwin14/share/julia/compiled/v1.12/REPL/u0gqU_vMi9E.dylib (unknown line)
jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-12/src/./julia.h:2391 [inlined]
start_task at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-12/src/task.c:1249
Allocations: 17932670 (Pool: 17932429; Big: 241); GC: 40
zsh: segmentation fault julia -tauto,1
