JuliaPy
Julia/CondaPkg.jl doesn't install a Julia package required by Python dependency
Big fan of PythonCall.jl, JuliaCall.jl, etc.. Thanks so much for these packages.
In my application, I have a chain of tools like this:
- OpenMDAOCore.jl is a pure-Julia package, with no dependencies other than Julia itself.
- omjlcomps is a Python package that depends on OpenMDAOCore.jl. I use juliapkg to specify that dependency via a juliapkg.json file: https://github.com/byuflowlab/OpenMDAO.jl/blob/master/python/omjlcomps/juliapkg.json
- OpenMDAO.jl is a Julia package that depends on both OpenMDAOCore.jl and omjlcomps. The CondaPkg.toml file declares omjlcomps as a dependency: https://github.com/byuflowlab/OpenMDAO.jl/blob/master/julia/OpenMDAO.jl/CondaPkg.toml
The problem: if I create a clean Julia environment and try to install OpenMDAO.jl, I get an error when I attempt to import OpenMDAO.jl. For some reason omjlcomps can't find OpenMDAOCore.jl. Here's what that looks like:
conda-forge/linux-64 32.1MB @ 2.9MB/s 11.5s
Transaction
Prefix: /tmp/jl_qAPRx5/.CondaPkg/env
Updating specs:
- conda-forge::libstdcxx-ng[version='>=3.4,<13.0']
- openmdao[version='>=3.26.0,<4']
- pip[version='>=22.0.0']
- conda-forge::python[version='>=3.7,<4',build=*cpython*]
Package Version Build Channel Size
────────────────────────────────────────────────────────────────────────────────────────
Install:
────────────────────────────────────────────────────────────────────────────────────────
+ _libgcc_mutex 0.1 conda_forge conda-forge/linux-64 Cached
+ _openmp_mutex 4.5 2_gnu conda-forge/linux-64 Cached
+ brotli 1.0.9 h166bdaf_8 conda-forge/linux-64 Cached
+ brotli-bin 1.0.9 h166bdaf_8 conda-forge/linux-64 Cached
+ bzip2 1.0.8 h7f98852_4 conda-forge/linux-64 Cached
+ ca-certificates 2023.5.7 hbcca054_0 conda-forge/linux-64 Cached
+ certifi 2023.5.7 pyhd8ed1ab_0 conda-forge/noarch Cached
+ charset-normalizer 3.1.0 pyhd8ed1ab_0 conda-forge/noarch Cached
+ idna 3.4 pyhd8ed1ab_0 conda-forge/noarch Cached
+ ld_impl_linux-64 2.40 h41732ed_0 conda-forge/linux-64 Cached
+ libblas 3.9.0 17_linux64_openblas conda-forge/linux-64 Cached
+ libbrotlicommon 1.0.9 h166bdaf_8 conda-forge/linux-64 Cached
+ libbrotlidec 1.0.9 h166bdaf_8 conda-forge/linux-64 Cached
+ libbrotlienc 1.0.9 h166bdaf_8 conda-forge/linux-64 Cached
+ libcblas 3.9.0 17_linux64_openblas conda-forge/linux-64 Cached
+ libexpat 2.5.0 hcb278e6_1 conda-forge/linux-64 Cached
+ libffi 3.4.2 h7f98852_5 conda-forge/linux-64 Cached
+ libgcc-ng 13.1.0 he5830b7_0 conda-forge/linux-64 Cached
+ libgfortran-ng 13.1.0 h69a702a_0 conda-forge/linux-64 Cached
+ libgfortran5 13.1.0 h15d22d2_0 conda-forge/linux-64 Cached
+ libgomp 13.1.0 he5830b7_0 conda-forge/linux-64 Cached
+ liblapack 3.9.0 17_linux64_openblas conda-forge/linux-64 Cached
+ libnsl 2.0.0 h7f98852_0 conda-forge/linux-64 Cached
+ libopenblas 0.3.23 pthreads_h80387f5_0 conda-forge/linux-64 Cached
+ libsqlite 3.42.0 h2797004_0 conda-forge/linux-64 Cached
+ libstdcxx-ng 12.3.0 h0f45ef3_0 conda-forge/linux-64 Cached
+ libuuid 2.38.1 h0b41bf4_0 conda-forge/linux-64 Cached
+ libzlib 1.2.13 hd590300_5 conda-forge/linux-64 Cached
+ ncurses 6.4 hcb278e6_0 conda-forge/linux-64 Cached
+ networkx 3.1 pyhd8ed1ab_0 conda-forge/noarch Cached
+ numpy 1.24.3 py311h64a7726_0 conda-forge/linux-64 Cached
+ openmdao 3.26.0 pyhd8ed1ab_0 conda-forge/noarch Cached
+ openssl 3.1.1 hd590300_1 conda-forge/linux-64 Cached
+ packaging 23.1 pyhd8ed1ab_0 conda-forge/noarch Cached
+ pip 23.1.2 pyhd8ed1ab_0 conda-forge/noarch Cached
+ platformdirs 3.5.3 pyhd8ed1ab_0 conda-forge/noarch Cached
+ pooch 1.7.0 pyha770c72_3 conda-forge/noarch Cached
+ pysocks 1.7.1 pyha2e5f31_6 conda-forge/noarch Cached
+ python 3.11.4 hab00c5b_0_cpython conda-forge/linux-64 Cached
+ python_abi 3.11 3_cp311 conda-forge/linux-64 Cached
+ readline 8.2 h8228510_1 conda-forge/linux-64 Cached
+ requests 2.31.0 pyhd8ed1ab_0 conda-forge/noarch Cached
+ scipy 1.10.1 py311h64a7726_3 conda-forge/linux-64 Cached
+ setuptools 67.7.2 pyhd8ed1ab_0 conda-forge/noarch Cached
+ tk 8.6.12 h27826a3_0 conda-forge/linux-64 Cached
+ typing-extensions 4.6.3 hd8ed1ab_0 conda-forge/noarch Cached
+ typing_extensions 4.6.3 pyha770c72_0 conda-forge/noarch Cached
+ tzdata 2023c h71feb2d_0 conda-forge/noarch Cached
+ urllib3 2.0.3 pyhd8ed1ab_0 conda-forge/noarch Cached
+ wheel 0.40.0 pyhd8ed1ab_0 conda-forge/noarch Cached
+ xz 5.2.6 h166bdaf_0 conda-forge/linux-64 Cached
Summary:
Install: 51 packages
Total download: 0 B
────────────────────────────────────────────────────────────────────────────────────────
Transaction starting
Linking libstdcxx-ng-12.3.0-h0f45ef3_0
Linking _libgcc_mutex-0.1-conda_forge
Linking libgfortran5-13.1.0-h15d22d2_0
Linking python_abi-3.11-3_cp311
Linking ld_impl_linux-64-2.40-h41732ed_0
Linking ca-certificates-2023.5.7-hbcca054_0
Linking libgomp-13.1.0-he5830b7_0
Linking libgfortran-ng-13.1.0-h69a702a_0
Linking _openmp_mutex-4.5-2_gnu
Linking libgcc-ng-13.1.0-he5830b7_0
Linking libbrotlicommon-1.0.9-h166bdaf_8
Linking libopenblas-0.3.23-pthreads_h80387f5_0
Linking openssl-3.1.1-hd590300_1
Linking libzlib-1.2.13-hd590300_5
Linking libffi-3.4.2-h7f98852_5
Linking bzip2-1.0.8-h7f98852_4
Linking ncurses-6.4-hcb278e6_0
Linking libuuid-2.38.1-h0b41bf4_0
Linking libexpat-2.5.0-hcb278e6_1
Linking xz-5.2.6-h166bdaf_0
Linking libnsl-2.0.0-h7f98852_0
Linking libbrotlienc-1.0.9-h166bdaf_8
Linking libbrotlidec-1.0.9-h166bdaf_8
Linking libblas-3.9.0-17_linux64_openblas
Linking libsqlite-3.42.0-h2797004_0
Linking tk-8.6.12-h27826a3_0
Linking readline-8.2-h8228510_1
Linking brotli-bin-1.0.9-h166bdaf_8
Linking libcblas-3.9.0-17_linux64_openblas
Linking liblapack-3.9.0-17_linux64_openblas
Linking brotli-1.0.9-h166bdaf_8
Linking tzdata-2023c-h71feb2d_0
Linking python-3.11.4-hab00c5b_0_cpython
Linking wheel-0.40.0-pyhd8ed1ab_0
Linking setuptools-67.7.2-pyhd8ed1ab_0
Linking pip-23.1.2-pyhd8ed1ab_0
Linking typing_extensions-4.6.3-pyha770c72_0
Linking charset-normalizer-3.1.0-pyhd8ed1ab_0
Linking pysocks-1.7.1-pyha2e5f31_6
Linking idna-3.4-pyhd8ed1ab_0
Linking certifi-2023.5.7-pyhd8ed1ab_0
Linking packaging-23.1-pyhd8ed1ab_0
Linking networkx-3.1-pyhd8ed1ab_0
Linking typing-extensions-4.6.3-hd8ed1ab_0
Linking urllib3-2.0.3-pyhd8ed1ab_0
Linking platformdirs-3.5.3-pyhd8ed1ab_0
Linking requests-2.31.0-pyhd8ed1ab_0
Linking pooch-1.7.0-pyha770c72_3
Linking numpy-1.24.3-py311h64a7726_0
Linking scipy-1.10.1-py311h64a7726_3
Linking openmdao-3.26.0-pyhd8ed1ab_0
Transaction finished
CondaPkg Installing Pip packages
│ /tmp/jl_qAPRx5/.CondaPkg/env/bin/pip
│ install
│ juliapkg ~=0.1.10
└ omjlcomps ~=0.2.0
Collecting juliapkg~=0.1.10
Using cached juliapkg-0.1.10-py3-none-any.whl (15 kB)
Collecting omjlcomps~=0.2.0
Using cached omjlcomps-0.2.3-py3-none-any.whl (13 kB)
Collecting semantic-version~=2.9 (from juliapkg~=0.1.10)
Using cached semantic_version-2.10.0-py2.py3-none-any.whl (15 kB)
Requirement already satisfied: openmdao~=3.26.0 in /tmp/jl_qAPRx5/.CondaPkg/env/lib/python3.11/site-packages (from omjlcomps~=0.2.0) (3.26.0)
Collecting juliacall~=0.9.13 (from omjlcomps~=0.2.0)
Using cached juliacall-0.9.13-py3-none-any.whl (11 kB)
Requirement already satisfied: networkx>=2.0 in /tmp/jl_qAPRx5/.CondaPkg/env/lib/python3.11/site-packages (from openmdao~=3.26.0->omjlcomps~=0.2.0) (3.1)
Requirement already satisfied: numpy in /tmp/jl_qAPRx5/.CondaPkg/env/lib/python3.11/site-packages (from openmdao~=3.26.0->omjlcomps~=0.2.0) (1.24.3)
Requirement already satisfied: scipy in /tmp/jl_qAPRx5/.CondaPkg/env/lib/python3.11/site-packages (from openmdao~=3.26.0->omjlcomps~=0.2.0) (1.10.1)
Requirement already satisfied: requests in /tmp/jl_qAPRx5/.CondaPkg/env/lib/python3.11/site-packages (from openmdao~=3.26.0->omjlcomps~=0.2.0) (2.31.0)
Requirement already satisfied: packaging in /tmp/jl_qAPRx5/.CondaPkg/env/lib/python3.11/site-packages (from openmdao~=3.26.0->omjlcomps~=0.2.0) (23.1)
Requirement already satisfied: charset-normalizer<4,>=2 in /tmp/jl_qAPRx5/.CondaPkg/env/lib/python3.11/site-packages (from requests->openmdao~=3.26.0->omjlcomps~=0.2.0) (3.1.0)
Requirement already satisfied: idna<4,>=2.5 in /tmp/jl_qAPRx5/.CondaPkg/env/lib/python3.11/site-packages (from requests->openmdao~=3.26.0->omjlcomps~=0.2.0) (3.4)
Requirement already satisfied: urllib3<3,>=1.21.1 in /tmp/jl_qAPRx5/.CondaPkg/env/lib/python3.11/site-packages (from requests->openmdao~=3.26.0->omjlcomps~=0.2.0) (2.0.3)
Requirement already satisfied: certifi>=2017.4.17 in /tmp/jl_qAPRx5/.CondaPkg/env/lib/python3.11/site-packages (from requests->openmdao~=3.26.0->omjlcomps~=0.2.0) (2023.5.7)
Installing collected packages: semantic-version, juliapkg, juliacall, omjlcomps
Successfully installed juliacall-0.9.13 juliapkg-0.1.10 omjlcomps-0.2.3 semantic-version-2.10.0
ERROR: InitError: Python: Julia: ArgumentError: Package OpenMDAOCore not found in current path.
- Run `import Pkg; Pkg.add("OpenMDAOCore")` to install the OpenMDAOCore package.
Stacktrace:
[1] macro expansion
@ ./loading.jl:1163 [inlined]
[2] macro expansion
@ ./lock.jl:223 [inlined]
[3] require(into::Module, mod::Symbol)
@ Base ./loading.jl:1144
[4] eval
@ ./boot.jl:368 [inlined]
[5] eval
@ ./Base.jl:65 [inlined]
[6] pyjlmodule_seval(self::Module, expr::PythonCall.Py)
@ PythonCall ~/.julia/packages/PythonCall/1f5yE/src/jlwrap/module.jl:13
[7] _pyjl_callmethod(f::Any, self_::Ptr{PythonCall.C.PyObject}, args_::Ptr{PythonCall.C.PyObject}, nargs::Int64)
@ PythonCall ~/.julia/packages/PythonCall/1f5yE/src/jlwrap/base.jl:62
[8] _pyjl_callmethod(o::Ptr{PythonCall.C.PyObject}, args::Ptr{PythonCall.C.PyObject})
@ PythonCall.C ~/.julia/packages/PythonCall/1f5yE/src/cpython/jlwrap.jl:47
[9] PyImport_Import
@ ~/.julia/packages/PythonCall/1f5yE/src/cpython/pointers.jl:299 [inlined]
[10] pyimport(m::String)
@ PythonCall ~/.julia/packages/PythonCall/1f5yE/src/concrete/import.jl:11
[11] __init__()
@ OpenMDAO ~/projects/pythoncall_openmdao/dev/OpenMDAO.jl-clean/julia/OpenMDAO.jl/src/OpenMDAO.jl:14
[12] _include_from_serialized(pkg::Base.PkgId, path::String, depmods::Vector{Any})
@ Base ./loading.jl:831
[13] _tryrequire_from_serialized(pkg::Base.PkgId, path::String)
@ Base ./loading.jl:978
[14] _require(pkg::Base.PkgId)
@ Base ./loading.jl:1347
[15] _require_prelocked(uuidkey::Base.PkgId)
@ Base ./loading.jl:1200
[16] macro expansion
@ ./loading.jl:1180 [inlined]
[17] macro expansion
@ ./lock.jl:223 [inlined]
[18] require(into::Module, mod::Symbol)
@ Base ./loading.jl:1144
[19] eval
@ ./boot.jl:368 [inlined]
[20] eval_user_input(ast::Any, backend::REPL.REPLBackend)
@ REPL ~/local/julia/1.8.5/share/julia/stdlib/v1.8/REPL/src/REPL.jl:151
[21] repl_backend_loop(backend::REPL.REPLBackend)
@ REPL ~/local/julia/1.8.5/share/julia/stdlib/v1.8/REPL/src/REPL.jl:247
[22] start_repl_backend(backend::REPL.REPLBackend, consumer::Any)
@ REPL ~/local/julia/1.8.5/share/julia/stdlib/v1.8/REPL/src/REPL.jl:232
[23] run_repl(repl::REPL.AbstractREPL, consumer::Any; backend_on_current_task::Bool)
@ REPL ~/local/julia/1.8.5/share/julia/stdlib/v1.8/REPL/src/REPL.jl:369
[24] run_repl(repl::REPL.AbstractREPL, consumer::Any)
@ REPL ~/local/julia/1.8.5/share/julia/stdlib/v1.8/REPL/src/REPL.jl:355
[25] (::Base.var"#967#969"{Bool, Bool, Bool})(REPL::Module)
@ Base ./client.jl:419
[26] #invokelatest#2
@ ./essentials.jl:729 [inlined]
[27] invokelatest
@ ./essentials.jl:726 [inlined]
[28] run_main_repl(interactive::Bool, quiet::Bool, banner::Bool, history_file::Bool, color_set::Bool)
@ Base ./client.jl:404
[29] exec_options(opts::Base.JLOptions)
@ Base ./client.jl:318
[30] _start()
@ Base ./client.jl:522
Python stacktrace:
[1] seval
@ ~/.julia/packages/PythonCall/1f5yE/src/jlwrap/module.jl:25
[2] <module>
@ /tmp/jl_qAPRx5/.CondaPkg/env/lib/python3.11/site-packages/omjlcomps/__init__.py:9
Stacktrace:
[1] pythrow()
@ PythonCall ~/.julia/packages/PythonCall/1f5yE/src/err.jl:94
[2] errcheck
@ ~/.julia/packages/PythonCall/1f5yE/src/err.jl:10 [inlined]
[3] pyimport(m::String)
@ PythonCall ~/.julia/packages/PythonCall/1f5yE/src/concrete/import.jl:11
[4] __init__()
@ OpenMDAO ~/projects/pythoncall_openmdao/dev/OpenMDAO.jl-clean/julia/OpenMDAO.jl/src/OpenMDAO.jl:14
[5] _include_from_serialized(pkg::Base.PkgId, path::String, depmods::Vector{Any})
@ Base ./loading.jl:831
[6] _tryrequire_from_serialized(pkg::Base.PkgId, path::String)
@ Base ./loading.jl:978
[7] _require(pkg::Base.PkgId)
@ Base ./loading.jl:1347
[8] _require_prelocked(uuidkey::Base.PkgId)
@ Base ./loading.jl:1200
[9] macro expansion
@ ./loading.jl:1180 [inlined]
[10] macro expansion
@ ./lock.jl:223 [inlined]
[11] require(into::Module, mod::Symbol)
@ Base ./loading.jl:1144
during initialization of module OpenMDAO
julia>
You also might be able to see the error from the AutoMerge workflow for the Julia General registry: https://github.com/JuliaRegistries/General/actions/runs/5244999369/jobs/9471800986?pr=85376.
I was able to work around this by just explicitly adding OpenMDAOCore.jl using Pkg in OpenMDAO.jl's __init__():
https://github.com/byuflowlab/OpenMDAO.jl/blob/892afe54b78dca25a6b66237ed24a584f5dc1c47/julia/OpenMDAO.jl/src/OpenMDAO.jl#L15
But I wonder if I'm doing something wrong, or if there's a better way?
Yeah CondaPkg/JuliaPkg doesn't support recursive Julia -> Python -> Julia dependencies. It's too hard.
I think the best solution right now is to add OpenMDAOCore as a dependency to OpenMDAO.
Gotcha, thanks. To be clear, OpenMDAOCore.jl was always a dependency of OpenMDAO.jl. The error occurs when installing OpenMDAO.jl in an environment where OpenMDAOCore.jl isn't explicitly installed (i.e., where OpenMDAOCore.jl isn't in the environment's Project.toml). The solution I came up with was adding this to OpenMDAO.jl:
function __init__()
PythonCall.pycopy!(om, PythonCall.pyimport("openmdao.api"))
Pkg.add("OpenMDAOCore")
PythonCall.pycopy!(omjlcomps, PythonCall.pyimport("omjlcomps"))
end
which I think installs OpenMDAOCore.jl in the current environment just before attempting to import the python package omjlcomps.
I'm fine with that solution, just want to make sure that's what you're suggesting.
Oh I see! Even though OpenMDAOCore is a dep of OpenMDAO, if you install OpenMDAO into some other project, then import OpenMDAOCore fails because it's not a direct dependency in the project. That is annoying. I'm not sure there's anything I could change in PythonCall to make this work, but will have a think.
You could use Base.require to import OpenMDAOCore instead. This circumvents these checks.
Edit: I'd recommend against stuff like calling Pkg from __init__.
Also, is it important to have these 3 separate packages. Could you instead have one OpenMDAO package with all of this stuff in?
You could use Base.require to import OpenMDAOCore instead. This circumvents these checks.
Interesting, I haven't heard of Base.require. How would I do that? I see it takes a module and a symbol (the name of the module I want to load, :OpenMDAOCore in this case?) https://docs.julialang.org/en/v1/base/base/#Base.require. What would the first argument be, and where would I put it? At the top of OpenMDAO.jl?
Also, is it important to have these 3 separate packages. Could you instead have one OpenMDAO package with all of this stuff in?
That's how it was set up originally, but it wasn't great for a couple of reasons. First, I think putting it all in one package would lead to a circular dependency:
- The Python package
omjlcompsneeds access to the methods defined in OpenMDAOCore.jl, so it's a dependency. - The Julia package OpenMDAO.jl needs the classes defined in the Python package
omjlcomps.
So if all the Julia code was in just one package, omjlcomps would need OpenMDAO.jl, and OpenMDAO.jl would need omjlcomps. That actually seemed to work, though I would get scary warnings re: the OpenMDAO.jl module being loaded twice.
Also, with the way things are right now, OpenMDAOCore.jl is a very lightweight dependency—actually has no dependencies other than Julia itself. So it allows Julia package developers to support OpenMDAO without messing around with the omjlcomps Python package, or Python at all, which is nice.
All valid reasons.
I think you can do something like
Base.require(Base.PkgId("name", "uuid"))
to import a specific package.
Gotcha, I tried
Base.require(Base.PkgId(Base.UUID("24d19c10-6eee-420f-95df-4537264b2753"), "OpenMDAOCore"))
at both the top level of the "parent" package OpenMDAO and inside OpenMDAO's __init__(), but still get the same error.