PythonCall.jl
PythonCall.jl copied to clipboard
JuliaPy
`ERROR: Python: ImportError: libspqr.so.2: undefined symbol: cholmod_l_metis`
Python
$ /cvmfs/icecube.opensciencegrid.org/py3-v4.1.1/RHEL_7_x86_64/bin/python3
Python 3.7.5 (default, Jul 24 2020, 21:48:49)
[GCC 9.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import photospline
>>>
PythonCall.jl
$ env JULIA_CONDAPKG_BACKEND=Null JULIA_PYTHONCALL_EXE=/cvmfs/icecube.opensciencegrid.org/py3-v4.1.1/RHEL_7_x86_64/bin/python3 julia
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.9.4 (2023-11-14)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
julia> using PythonCall
julia> pyimport("math")
Python: <module 'math' from '/cvmfs/icecube.opensciencegrid.org/py3-v4.1.1/RHEL_7_x86_64/lib/python3.7/lib-dynload/math.cpython-37m-x86_64-linux-gnu.so'>
julia> pyimport("photospline")
ERROR: Python: ImportError: /cvmfs/icecube.opensciencegrid.org/py3-v4.1.1/RHEL_7_x86_64/spack/opt/spack/linux-centos7-x86_64/gcc-9.2.0spack/suite-sparse-5.2.0-d7wcgqk77zldo6ndrjuvujv2xvgudsel/lib/libspqr.so.2: undefined symbol: cholmod_l_metis
Python stacktrace: none
Stacktrace:
[1] pythrow()
@ PythonCall ~/.julia/packages/PythonCall/wXfah/src/err.jl:94
[2] errcheck
@ ~/.julia/packages/PythonCall/wXfah/src/err.jl:10 [inlined]
[3] pyimport(m::String)
@ PythonCall ~/.julia/packages/PythonCall/wXfah/src/concrete/import.jl:11
[4] top-level scope
@ REPL[3]:1
this problem exists in 1.8 and 1.9 but seems to be fixed in 1.10-rc1
I know this with some certainty because the same problem also exists when using Julia-managed Conda environment (in 1.8 and 1.9)
julia> using PythonCall
julia> pyimport("photospline")
ERROR: Python: ImportError: /home/akako/.conda/envs/hep/lib/python3.11/site-packages/../.././libspqr.so.2: undefined symbol: cholmod_l_metis
Python stacktrace: none
Stacktrace:
[1] pythrow()
@ PythonCall ~/.julia/packages/PythonCall/wXfah/src/err.jl:94
[2] errcheck
@ ~/.julia/packages/PythonCall/wXfah/src/err.jl:10 [inlined]
[3] pyimport(m::String)
@ PythonCall ~/.julia/packages/PythonCall/wXfah/src/concrete/import.jl:11
[4] top-level scope
@ REPL[4]:1
From the name of the missing symbol I'm guessing it's a version clash between Conda and Julia for some linear algebra library?
Yeah my guess is Julia 1.9 uses a older linear algebra library than one of your Conda dependencies but Julia 1.10 uses a newer one. So in the future if you upgrade one and not the other they may clash again.
This is a fundamental issue with using two essentially independent package managers to install binaries which get loaded into the same process.
CondaPkg works around this for libstdc++ by detecting which version Julia is using and installing a compatible version of the corresponding Conda package. If you can figure out which library is causing your issue and the corresponding Conda package we might be able to fix it.
All assuming my hunch is correct!
I'm not loading any Julia package. If your theory is correct (high chance I think), then it makes sense because right now Julia loads LinearAlgebra.jl automatically in 1.9
Yeah did LinearAlgebra get removed from the sysimage or something in 1.10? That would explain it. Try loading LinearAlgebra in 1.10 before PythonCall and see if you see the issue again.
Just tried that, still works on 1.10, and I think 1.10 (or anywhere really) hasn't successfully excised LA.jl anyway:
- https://github.com/JuliaLang/julia/pull/51795
Looks like Julia went from SuiteSparse v5 to v7 between 1.9 and 1.10. What version is in your Conda environment?
https://github.com/icecube/photospline
no idea, I think whatever is newest on Conda when it was built?
(jl_sgJk2z) pkg> conda run conda list
ERROR: Can not run conda command when backend is Null. Manage conda actions outside of julia.
(jl_sgJk2z) pkg>
akako@frame ~> conda list (hep)
# packages in environment at /home/akako/.conda/envs/hep:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
aiofiles 22.1.0 py311h06a4308_0
aiosqlite 0.18.0 py311h06a4308_0
anyio 3.6.2 pypi_0 pypi
argon2-cffi 21.3.0 pyhd3eb1b0_0
argon2-cffi-bindings 21.2.0 py311h5eee18b_0
arrow 1.2.3 pypi_0 pypi
asttokens 2.2.1 pypi_0 pypi
attrs 23.1.0 pypi_0 pypi
awkward 2.2.0 pypi_0 pypi
awkward-cpp 15 pypi_0 pypi
babel 2.11.0 py311h06a4308_0
backcall 0.2.0 pyhd3eb1b0_0
beautifulsoup4 4.12.2 py311h06a4308_0
bleach 6.0.0 pypi_0 pypi
boost-histogram 1.4.0 pypi_0 pypi
brotlipy 0.7.0 py311h5eee18b_1002
bzip2 1.0.8 h7b6447c_0
c-ares 1.22.1 hd590300_0 conda-forge
ca-certificates 2023.11.17 hbcca054_0 conda-forge
certifi 2023.11.17 pyhd8ed1ab_0 conda-forge
cffi 1.15.1 py311h5eee18b_3
cfitsio 4.3.1 hbdc6101_0 conda-forge
charset-normalizer 2.0.4 pyhd3eb1b0_0
clean-text 0.4.0 pypi_0 pypi
click 8.1.6 pypi_0 pypi
comm 0.1.3 pypi_0 pypi
contourpy 1.1.0 pypi_0 pypi
cryptography 41.0.7 py311h63ff55d_0 conda-forge
cycler 0.11.0 pypi_0 pypi
dbus 1.13.18 hb2f20db_0
debugpy 1.6.7 pypi_0 pypi
decorator 5.1.1 pyhd3eb1b0_0
defusedxml 0.7.1 pyhd3eb1b0_0
emoji 2.6.0 pypi_0 pypi
entrypoints 0.4 py311h06a4308_0
executing 1.2.0 pypi_0 pypi
expat 2.4.9 h6a678d5_0
faker 19.6.2 pypi_0 pypi
fast-histogram 0.12 pypi_0 pypi
fastjsonschema 2.16.3 pypi_0 pypi
fontconfig 2.14.1 h4c34cd2_2
fonttools 4.40.0 pypi_0 pypi
fqdn 1.5.1 pypi_0 pypi
freetype 2.12.1 h4a9f257_0
fsspec 2023.6.0 pypi_0 pypi
ftfy 6.1.1 pypi_0 pypi
giflib 5.2.1 h5eee18b_3
glib 2.69.1 he621ea3_2
gmp 6.3.0 h59595ed_0 conda-forge
gst-plugins-base 1.14.1 h6a678d5_1
gstreamer 1.14.1 h5eee18b_1
hist 2.7.2 pypi_0 pypi
histoprint 2.4.0 pypi_0 pypi
icu 58.2 he6710b0_3
idna 3.4 py311h06a4308_0
install 1.3.5 pypi_0 pypi
ipykernel 6.23.0 pypi_0 pypi
ipython 8.13.2 pypi_0 pypi
ipython_genutils 0.2.0 pyhd3eb1b0_1
ipywidgets 8.0.6 pypi_0 pypi
isoduration 20.11.0 pypi_0 pypi
jedi 0.18.2 pypi_0 pypi
jinja2 3.1.2 py311h06a4308_0
jpeg 9e h5eee18b_1
json5 0.9.6 pyhd3eb1b0_0
jsonpatch 1.33 pypi_0 pypi
jsonpointer 2.3 pypi_0 pypi
jsonschema 4.17.3 py311h06a4308_0
julia 0.6.1 dev_0 <develop>
juliapkg 0.1.10 pypi_0 pypi
jupyter 1.0.0 py311h06a4308_8
jupyter-client 8.2.0 pypi_0 pypi
jupyter-contrib-core 0.4.2 pypi_0 pypi
jupyter-contrib-nbextensions 0.7.0 pypi_0 pypi
jupyter-core 5.3.1 pypi_0 pypi
jupyter-highlight-selected-word 0.2.0 pypi_0 pypi
jupyter-nbextensions-configurator 0.6.3 pypi_0 pypi
jupyter_client 8.1.0 py311h06a4308_0
jupyter_console 6.6.3 py311h06a4308_0
jupyter_events 0.6.3 py311h06a4308_0
jupyter_server 2.5.0 py311h06a4308_0
jupyter_server_fileid 0.9.0 py311h06a4308_0
jupyter_server_terminals 0.4.4 py311h06a4308_1
jupyter_server_ydoc 0.8.0 py311h06a4308_1
jupyter_ydoc 0.2.4 py311h06a4308_0
jupyterlab 3.6.3 py311h06a4308_0
jupyterlab-pygments 0.2.2 pypi_0 pypi
jupyterlab-rise 0.2.0 pypi_0 pypi
jupyterlab-widgets 3.0.7 pypi_0 pypi
jupyterlab_pygments 0.1.2 py_0
jupyterlab_server 2.22.0 py311h06a4308_0
jupyterlab_widgets 3.0.5 py311h06a4308_0
keyutils 1.6.1 h166bdaf_0 conda-forge
kiwisolver 1.4.4 pypi_0 pypi
krb5 1.20.1 h81ceb04_0 conda-forge
ld_impl_linux-64 2.38 h1181459_1
lerc 3.0 h295c915_0
libblas 3.9.0 20_linux64_openblas conda-forge
libcblas 3.9.0 20_linux64_openblas conda-forge
libclang 14.0.6 default_hc6dbbc7_1
libclang13 14.0.6 default_he11475f_1
libcups 2.3.3 h36d4200_3 conda-forge
libcurl 8.4.0 h251f7ec_0
libdeflate 1.17 h5eee18b_0
libedit 3.1.20221030 h5eee18b_0
libev 4.33 h516909a_1 conda-forge
libevent 2.1.12 hf998b51_1 conda-forge
libffi 3.4.4 h6a678d5_0
libgcc-ng 13.2.0 h807b86a_3 conda-forge
libgfortran-ng 13.2.0 h69a702a_3 conda-forge
libgfortran5 13.2.0 ha4646dd_3 conda-forge
libgomp 13.2.0 h807b86a_3 conda-forge
libhwloc 2.9.1 hd6dc26d_0 conda-forge
liblapack 3.9.0 20_linux64_openblas conda-forge
libllvm14 14.0.6 hdb19cb5_3
libnghttp2 1.58.0 h47da74e_0 conda-forge
libopenblas 0.3.25 pthreads_h413a1c8_0 conda-forge
libpng 1.6.39 h5eee18b_0
libpq 12.15 hdbd6064_1
libsodium 1.0.18 h7b6447c_0
libssh2 1.11.0 h0841786_0 conda-forge
libstdcxx-ng 13.2.0 h7e041cc_3 conda-forge
libtiff 4.5.0 h6a678d5_2
libuuid 1.41.5 h5eee18b_0
libwebp 1.2.4 h11a3e52_1
libwebp-base 1.2.4 h5eee18b_1
libxcb 1.15 h7f8727e_0
libxkbcommon 1.0.1 h5eee18b_1
libxml2 2.10.3 hcbfbd50_0
libxslt 1.1.37 h2085143_0
libzlib 1.2.13 hd590300_5 conda-forge
lxml 4.9.2 py311h5eee18b_0
lz4 4.3.2 pypi_0 pypi
lz4-c 1.9.4 h6a678d5_0
markupsafe 2.1.2 pypi_0 pypi
matplotlib 3.7.1 pypi_0 pypi
matplotlib-inline 0.1.6 py311h06a4308_0
metis 5.1.0 h59595ed_1007 conda-forge
mistune 3.0.1 pypi_0 pypi
mpfr 4.2.1 h9458935_0 conda-forge
mysql 5.7.20 hf484d3e_1001 conda-forge
nanoid 2.0.0 pypi_0 pypi
nbclassic 1.0.0 pypi_0 pypi
nbclient 0.7.4 pypi_0 pypi
nbconvert 7.6.0 pypi_0 pypi
nbformat 5.8.0 pypi_0 pypi
ncurses 6.4 h6a678d5_0
nest-asyncio 1.5.6 py311h06a4308_0
notebook 6.5.4 py311h06a4308_0
notebook-shim 0.2.3 pypi_0 pypi
nspr 4.35 h6a678d5_0
nss 3.89.1 h6a678d5_0
numpy 1.24.3 pypi_0 pypi
openssl 3.2.0 hd590300_1 conda-forge
packaging 23.1 pypi_0 pypi
pandocfilters 1.5.0 pyhd3eb1b0_0
parso 0.8.3 pyhd3eb1b0_0
pcre 8.45 h295c915_0
pexpect 4.8.0 pyhd3eb1b0_3
photospline 2.2.1 py311h2453465_1 conda-forge
pickleshare 0.7.5 pyhd3eb1b0_1003
pillow 9.5.0 pypi_0 pypi
pip 23.0.1 py311h06a4308_0
platformdirs 3.5.1 pypi_0 pypi
ply 3.11 py311h06a4308_0
prometheus-client 0.16.0 pypi_0 pypi
prometheus_client 0.14.1 py311h06a4308_0
prompt-toolkit 3.0.38 pypi_0 pypi
prompt_toolkit 3.0.36 hd3eb1b0_0
psutil 5.9.5 pypi_0 pypi
ptyprocess 0.7.0 pyhd3eb1b0_2
pure_eval 0.2.2 pyhd3eb1b0_0
pyarrow 12.0.0 pypi_0 pypi
pycparser 2.21 pyhd3eb1b0_0
pygments 2.15.1 py311h06a4308_1
pyhf 0.7.2 pypi_0 pypi
pyopenssl 23.3.0 pyhd8ed1ab_0 conda-forge
pyparsing 3.1.0 pypi_0 pypi
pyqt 5.15.7 py311h6a678d5_0
pyqt5-sip 12.11.0 py311h6a678d5_0
pyrsistent 0.19.3 pypi_0 pypi
pysocks 1.7.1 py311h06a4308_0
python 3.11.5 h955ad1f_0
python-dateutil 2.8.2 pyhd3eb1b0_0
python-fastjsonschema 2.16.2 py311h06a4308_0
python-json-logger 2.0.7 py311h06a4308_0
python_abi 3.11 2_cp311 conda-forge
pytz 2022.7 py311h06a4308_0
pyyaml 6.0 py311h5eee18b_1
pyzmq 25.0.2 pypi_0 pypi
qt-main 5.15.2 h7358343_9
qt-webengine 5.15.9 hbbf29b9_6
qtconsole 5.4.3 pypi_0 pypi
qtpy 2.3.1 pypi_0 pypi
qtwebkit 5.212 h3fafdc1_5
readline 8.2 h5eee18b_0
requests 2.29.0 py311h06a4308_0
rfc3339-validator 0.1.4 py311h06a4308_0
rfc3986-validator 0.1.1 py311h06a4308_0
rise 5.7.1 pypi_0 pypi
scipy 1.11.1 pypi_0 pypi
semantic-version 2.10.0 pypi_0 pypi
send2trash 1.8.2 pypi_0 pypi
setuptools 66.0.0 py311h06a4308_0
sip 6.6.2 py311h6a678d5_0
six 1.16.0 pyhd3eb1b0_1
sniffio 1.3.0 pypi_0 pypi
soupsieve 2.4.1 pypi_0 pypi
sqlite 3.41.2 h5eee18b_0
stack-data 0.6.2 pypi_0 pypi
stack_data 0.2.0 pyhd3eb1b0_0
suitesparse 5.10.1 h9e50725_1 conda-forge
tbb 2021.9.0 hf52228f_0 conda-forge
terminado 0.17.1 py311h06a4308_0
tinycss2 1.2.1 py311h06a4308_0
tk 8.6.12 h1ccaba5_0
toml 0.10.2 pyhd3eb1b0_0
tornado 6.3.1 pypi_0 pypi
tqdm 4.65.0 pypi_0 pypi
traitlets 5.9.0 pypi_0 pypi
typing-extensions 4.6.3 py311h06a4308_0
typing_extensions 4.6.3 py311h06a4308_0
tzdata 2023c h04d1e81_0
uhi 0.4.0 pypi_0 pypi
unidecode 1.3.6 pypi_0 pypi
uproot 5.0.7 pypi_0 pypi
uri-template 1.2.0 pypi_0 pypi
urllib3 1.26.16 py311h06a4308_0
vector 1.1.0 pypi_0 pypi
wcwidth 0.2.6 pypi_0 pypi
webcolors 1.13 pypi_0 pypi
webencodings 0.5.1 pypi_0 pypi
websocket-client 1.5.1 pypi_0 pypi
wheel 0.38.4 py311h06a4308_0
widgetsnbextension 4.0.7 pypi_0 pypi
xxhash 3.2.0 pypi_0 pypi
xz 5.4.2 h5eee18b_0
y-py 0.5.9 py311h52d8a92_0
yaml 0.2.5 h7b6447c_0
ypy-websocket 0.8.2 py311h06a4308_0
zeromq 4.3.4 h2531618_0
zlib 1.2.13 hd590300_5 conda-forge
zstd 1.5.5 hc292b87_0