julia
julia copied to clipboard
JuliaLang
Broken backward compatibility with Juliacall
The new version 1.10.1 release broke the package juliacall on Linux systems. When importing the package from python a segmentation fault occurs.
How to reproduce:
- Have julia version 1.10.1 installed in your system (or no version at all, since juliacall installs julia if not present).
- create a clean conda environment with python (3.9 or superior tested).
- run
pip install juliacall - open python interactive session and run
import juliacall
Note: this does not happen with version 1.10.0 and on Mac devices.
I cannot reproduce this on asahi linux
x@fedora:~/Downloads$ python
Python 3.12.1 (main, Dec 18 2023, 00:00:00) [GCC 13.2.1 20231205 (Red Hat 13.2.1-6)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import juliacall
[juliapkg] Locating Julia ^1.6.1
[juliapkg] Using Julia 1.10.1 at /home/x/.julia/juliaup/julia-1.10.1+0.aarch64.linux.gnu/bin/julia
[juliapkg] Using Julia project at /home/x/.julia/environments/pyjuliapkg
[juliapkg] Installing packages:
julia> import Pkg
julia> Pkg.add([Pkg.PackageSpec(name="PythonCall", uuid="6099a3de-0909-46bc-b1f4-468b9a2dfc0d")])
julia> Pkg.resolve()
julia> Pkg.precompile()
Resolving package versions...
Updating `~/.julia/environments/pyjuliapkg/Project.toml`
[6099a3de] + PythonCall v0.9.15
Updating `~/.julia/environments/pyjuliapkg/Manifest.toml`
[992eb4ea] + CondaPkg v0.2.22
[9a962f9c] + DataAPI v1.16.0
[e2d170a0] + DataValueInterfaces v1.0.0
[82899510] + IteratorInterfaceExtensions v1.0.0
[692b3bcd] + JLLWrappers v1.5.0
[0f8b85d8] + JSON3 v1.14.0
[1914dd2f] + MacroTools v0.5.13
[0b3b1443] + MicroMamba v0.1.14
[bac558e1] + OrderedCollections v1.6.3
[69de0a69] + Parsers v2.8.1
[fa939f87] + Pidfile v1.3.0
[aea7be01] + PrecompileTools v1.2.0
[21216c6a] + Preferences v1.4.1
[6099a3de] + PythonCall v0.9.15
[ae029012] + Requires v1.3.0
[6c6a2e73] + Scratch v1.2.1
[856f2bd8] + StructTypes v1.10.0
[3783bdb8] + TableTraits v1.0.1
[bd369af6] + Tables v1.11.1
[e17b2a0c] + UnsafePointers v1.0.0
[f8abcde7] + micromamba_jll v1.4.9+0
[0dad84c5] + ArgTools v1.1.1
[56f22d72] + Artifacts
[2a0f44e3] + Base64
[ade2ca70] + Dates
[f43a241f] + Downloads v1.6.0
[7b1f6079] + FileWatching
[b77e0a4c] + InteractiveUtils
[4af54fe1] + LazyArtifacts
[b27032c2] + LibCURL v0.6.4
[76f85450] + LibGit2
[8f399da3] + Libdl
[37e2e46d] + LinearAlgebra
[56ddb016] + Logging
[d6f4376e] + Markdown
[a63ad114] + Mmap
[ca575930] + NetworkOptions v1.2.0
[44cfe95a] + Pkg v1.10.0
[de0858da] + Printf
[3fa0cd96] + REPL
[9a3f8284] + Random
[ea8e919c] + SHA v0.7.0
[9e88b42a] + Serialization
[6462fe0b] + Sockets
[fa267f1f] + TOML v1.0.3
[a4e569a6] + Tar v1.10.0
[8dfed614] + Test
[cf7118a7] + UUIDs
[4ec0a83e] + Unicode
[e66e0078] + CompilerSupportLibraries_jll v1.1.0+0
[deac9b47] + LibCURL_jll v8.4.0+0
[e37daf67] + LibGit2_jll v1.6.4+0
[29816b5a] + LibSSH2_jll v1.11.0+1
[c8ffd9c3] + MbedTLS_jll v2.28.2+1
[14a3606d] + MozillaCACerts_jll v2023.1.10
[4536629a] + OpenBLAS_jll v0.3.23+4
[83775a58] + Zlib_jll v1.2.13+1
[8e850b90] + libblastrampoline_jll v5.8.0+1
[8e850ede] + nghttp2_jll v1.52.0+1
[3f19e933] + p7zip_jll v17.4.0+2
Precompiling project...
6 dependencies successfully precompiled in 38 seconds. 19 already precompiled.
1 dependency had output during precompilation:
┌ micromamba_jll
│ Downloading artifact: micromamba
│
│ [pid 71524] waiting for IO to finish:
│ Handle type uv_handle_t->data
│ timer 0x40401550->0xfffec4500ac0
│ This means that a package has started a background task or event source that has not finished running. For precompilation to complete successfully, the event source needs to be closed explicitly. See the developer documentation on fixing precompilation hangs for more help.
│
│ [pid 71524] waiting for IO to finish:
│ Handle type uv_handle_t->data
│ timer 0x40401550->0xfffec4500ac0
│ This means that a package has started a background task or event source that has not finished running. For precompilation to complete successfully, the event source needs to be closed explicitly. See the developer documentation on fixing precompilation hangs for more help.
└
No Changes to `~/.julia/environments/pyjuliapkg/Project.toml`
No Changes to `~/.julia/environments/pyjuliapkg/Manifest.toml`
>>> juliacall.Main.VERSION
Julia: v"1.10.1"
>>> juliacall.Main.versioninfo()
Julia Version 1.10.1
Commit 7790d6f0641 (2024-02-13 20:41 UTC)
Build Info:
Official https://julialang.org/ release
Platform Info:
OS: Linux (aarch64-linux-gnu)
CPU: 8 × unknown
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-15.0.7 (ORCJIT, generic)
Threads: 1 default, 0 interactive, 1 GC (on 8 virtual cores)
Environment:
JULIA_PYTHONCALL_EXE = /usr/bin/python
Here is what I get when I try to reproduce the error:
Python 3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import juliacall
[juliapkg] Locating Julia ^1.6.1
[juliapkg] Installing Julia 1.10.1 using JuliaUp
[juliapkg] Querying Julia versions from https://julialang-s3.julialang.org/bin/versions.json
[juliapkg] WARNING: About to install Julia 1.10.1 to /home/brusa/miniconda3/envs/reproduce/julia_env/pyjuliapkg/install.
[juliapkg] If you use juliapkg in more than one environment, you are likely to have Julia
[juliapkg] installed in multiple locations. It is recommended to install JuliaUp
[juliapkg] (https://github.com/JuliaLang/juliaup) or Julia (https://julialang.org/downloads)
[juliapkg] yourself.
[juliapkg] Downloading Julia from https://julialang-s3.julialang.org/bin/linux/x64/1.10/julia-1.10.1-linux-x86_64.tar.gz
downloaded 87.5 MB of 160.1 MB
download complete
[juliapkg] Verifying download
[juliapkg] Installing Julia 1.10.1 to /home/brusa/miniconda3/envs/reproduce/julia_env/pyjuliapkg/install
[juliapkg] Using Julia 1.10.1 at /home/brusa/miniconda3/envs/reproduce/julia_env/pyjuliapkg/install/bin/julia
[juliapkg] Using Julia project at /home/brusa/miniconda3/envs/reproduce/julia_env
[juliapkg] Installing packages:
julia> import Pkg
julia> Pkg.add([Pkg.PackageSpec(name="PythonCall", uuid="6099a3de-0909-46bc-b1f4-468b9a2dfc0d")])
julia> Pkg.resolve()
julia> Pkg.precompile()
Resolving package versions...
Updating `~/miniconda3/envs/reproduce/julia_env/Project.toml`
[6099a3de] + PythonCall v0.9.15
Updating `~/miniconda3/envs/reproduce/julia_env/Manifest.toml`
[992eb4ea] + CondaPkg v0.2.22
[9a962f9c] + DataAPI v1.16.0
[e2d170a0] + DataValueInterfaces v1.0.0
[82899510] + IteratorInterfaceExtensions v1.0.0
[692b3bcd] + JLLWrappers v1.5.0
[0f8b85d8] + JSON3 v1.14.0
[1914dd2f] + MacroTools v0.5.13
[0b3b1443] + MicroMamba v0.1.14
[bac558e1] + OrderedCollections v1.6.3
[69de0a69] + Parsers v2.8.1
[fa939f87] + Pidfile v1.3.0
[aea7be01] + PrecompileTools v1.2.0
[21216c6a] + Preferences v1.4.1
[6099a3de] + PythonCall v0.9.15
[ae029012] + Requires v1.3.0
[6c6a2e73] + Scratch v1.2.1
[856f2bd8] + StructTypes v1.10.0
[3783bdb8] + TableTraits v1.0.1
[bd369af6] + Tables v1.11.1
[e17b2a0c] + UnsafePointers v1.0.0
[f8abcde7] + micromamba_jll v1.4.9+0
[0dad84c5] + ArgTools v1.1.1
[56f22d72] + Artifacts
[2a0f44e3] + Base64
[ade2ca70] + Dates
[f43a241f] + Downloads v1.6.0
[7b1f6079] + FileWatching
[b77e0a4c] + InteractiveUtils
[4af54fe1] + LazyArtifacts
[b27032c2] + LibCURL v0.6.4
[76f85450] + LibGit2
[8f399da3] + Libdl
[37e2e46d] + LinearAlgebra
[56ddb016] + Logging
[d6f4376e] + Markdown
[a63ad114] + Mmap
[ca575930] + NetworkOptions v1.2.0
[44cfe95a] + Pkg v1.10.0
[de0858da] + Printf
[3fa0cd96] + REPL
[9a3f8284] + Random
[ea8e919c] + SHA v0.7.0
[9e88b42a] + Serialization
[6462fe0b] + Sockets
[fa267f1f] + TOML v1.0.3
[a4e569a6] + Tar v1.10.0
[8dfed614] + Test
[cf7118a7] + UUIDs
[4ec0a83e] + Unicode
[e66e0078] + CompilerSupportLibraries_jll v1.1.0+0
[deac9b47] + LibCURL_jll v8.4.0+0
[e37daf67] + LibGit2_jll v1.6.4+0
[29816b5a] + LibSSH2_jll v1.11.0+1
[c8ffd9c3] + MbedTLS_jll v2.28.2+1
[14a3606d] + MozillaCACerts_jll v2023.1.10
[4536629a] + OpenBLAS_jll v0.3.23+4
[83775a58] + Zlib_jll v1.2.13+1
[8e850b90] + libblastrampoline_jll v5.8.0+1
[8e850ede] + nghttp2_jll v1.52.0+1
[3f19e933] + p7zip_jll v17.4.0+2
Precompiling project...
1 dependency successfully precompiled in 2 seconds. 24 already precompiled.
No Changes to `~/miniconda3/envs/reproduce/julia_env/Project.toml`
No Changes to `~/miniconda3/envs/reproduce/julia_env/Manifest.toml`
zsh: segmentation fault (core dumped) python
I have been able to reproduce it also on a Ubuntu VM.
Both devices have no GPU, here is the output of lscpu on my device:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 48 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Vendor ID: AuthenticAMD
Model name: AMD Ryzen 3 5300U with Radeon Graphics
CPU family: 23
Model: 104
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 1
Stepping: 1
Frequency boost: enabled
CPU(s) scaling MHz: 47%
CPU max MHz: 3900,0000
CPU min MHz: 1400,0000
BogoMIPS: 5192,50
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpu
id extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dn
owprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rd
t_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock nri
p_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca
Virtualization features:
Virtualization: AMD-V
Caches (sum of all):
L1d: 128 KiB (4 instances)
L1i: 128 KiB (4 instances)
L2: 2 MiB (4 instances)
L3: 4 MiB (1 instance)
NUMA:
NUMA node(s): 1
NUMA node0 CPU(s): 0-7
Vulnerabilities:
Gather data sampling: Not affected
Itlb multihit: Not affected
L1tf: Not affected
Mds: Not affected
Meltdown: Not affected
Mmio stale data: Not affected
Retbleed: Mitigation; untrained return thunk; SMT enabled with STIBP protection
Spec rstack overflow: Mitigation; safe RET
Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Spectre v2: Mitigation; Retpolines, IBPB conditional, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
Srbds: Not affected
Tsx async abort: Not affected
This feels similar to https://github.com/JuliaPy/PythonCall.jl/issues/215, but you have a clean environment, so it shouldn't be a pytorch-julia incompatibility.
@gbaraldi and I have run into a similar issue upon loading libgomp (which CompilerSupportLibraries_jll` loads) from within Python with RTLD_DEEPBIND. The reproducer looks something like this:
// test.c
#include <stdio.h>
extern char **environ;
__attribute__((constructor))
static void my_init() {
printf("%p \n", environ[0]); // libgomp expects that `environ` has been initialized by libc
}
# test.py
import ctypes, os
lib = ctypes.CDLL("./libtest.so", os.RTLD_DEEPBIND) # loading Julia-provided libgomp also segfaults
$ clang test.c -shared -o libtest.so
$ python3 test.py # segfaults for some versions of Python
Segmentation fault (core dumped)
The problem is that the environ symbol Python provides (if built without -fPIC) conflicts with the definition provided by glibc, and the resolved symbol is incorrect if loading with RTLD_DEEPBIND:
$ readelf -sW /usr/bin/python3 | grep environ
1622: 0000000000a92620 8 OBJECT WEAK DEFAULT 29 environ@GLIBC_2.2.5 (2)
1700: 0000000000a92620 8 OBJECT GLOBAL DEFAULT 29 __environ@GLIBC_2.2.5 (2)
$ readelf -sW /usr/lib/x86_64-linux-gnu/libc.so.6 | grep environ
133: 0000000000222200 8 OBJECT WEAK DEFAULT 35 _environ@@GLIBC_2.2.5
724: 0000000000222200 8 OBJECT GLOBAL DEFAULT 35 __environ@@GLIBC_2.2.5
958: 0000000000222200 8 OBJECT WEAK DEFAULT 35 environ@@GLIBC_2.2.5
This wasn't an issue before 1.10.1 because libgomp did not use environ until it was updated to 13.2.0
