JuliaPy
Trouble installing PyJulia
Python 3.7.6 | packaged by conda-forge | (default, Mar 23 2020, 23:03:20)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import julia
>>> julia.install()
[ Info: Julia version info
Julia Version 1.5.0-rc1.0
Commit 24f033c951 (2020-06-26 20:13 UTC)
Platform Info:
OS: Linux (x86_64-pc-linux-gnu)
"Fedora release 30 (Thirty)"
uname: Linux 5.3.7-200.fc30.x86_64 #1 SMP Fri Oct 18 20:13:59 UTC 2019 x86_64 x86_64
CPU: Intel(R) Xeon(R) Gold 6254 CPU @ 3.10GHz:
speed user nice sys idle irq
#1-18 3092 MHz 154700970 s 364803 s 58215826 s 2987251912 s 2277842 s
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-9.0.1 (ORCJIT, skylake)
Environment:
JULIA_NUM_THREADS = 12
JULIA_PKG_SERVER = pkg.julialang.org
JULIA_PROJECT = /home/x/foo/code/private/julia/foo
HOME = /home/x
MANPATH = :/opt/puppetlabs/puppet/share/man
PATH = /home/x/.nimble/bin:/home/x/.npm-global/bin:/home/x/bin:/home/x/node-v12.14.1-linux-x64/bin:/home/x/.nimble/bin:/home/x/miniconda3/envs/foo
/bin:/home/x/miniconda3/condabin:/home/x/.npm-global/bin:/home/x/bin:/home/x/node-v12.14.1-linux-x64/bin:/home/x/.local/bin:/home/x/bin:/usr/local/bin
:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin:/home/x/bin:/home/x/bin
PYTHONPATH = /home/x/foo/code/private/python:/home/x/foo/code/shared/python:/home/x/foo/code/private/python:/home/x/foo/code/shared/python:
TERM = screen
[ Info: Julia executable: /data/home/x/julia-1.5.0-rc1/bin/julia
[ Info: Trying to import PyCall...
┌ Warning: PyCall is already installed. However, you may have trouble using
│ this Python executable because it is statically linked to libpython.
│
│ For more information, see:
│ https://pyjulia.readthedocs.io/en/latest/troubleshooting.html
│
│ Python executable:
│ /home/x/miniconda3/envs/foo/bin/python
│ Julia executable:
│ /data/home/x/julia-1.5.0-rc1/bin/julia
└ @ Main /data/home/x/miniconda3/envs/foo/lib/python3.7/site-packages/julia/install.jl:74
(foo) ➜ code python3 -m julia.sysimage sys.so
Activating new environment at `/tmp/tmp.pyjulia.sysimage.f6s6rkax/compiler_env/Project.toml`
[ Info: Installing PackageCompiler...
Updating registry at `~/.julia/registries/General`
Resolving package versions...
Updating `/tmp/tmp.pyjulia.sysimage.f6s6rkax/compiler_env/Project.toml`
[9b87118b] + PackageCompiler v1.2.1
Updating `/tmp/tmp.pyjulia.sysimage.f6s6rkax/compiler_env/Manifest.toml`
[9b87118b] + PackageCompiler v1.2.1
[2a0f44e3] + Base64
[ade2ca70] + Dates
[b77e0a4c] + InteractiveUtils
[76f85450] + LibGit2
[8f399da3] + Libdl
[56ddb016] + Logging
[d6f4376e] + Markdown
[44cfe95a] + Pkg
[de0858da] + Printf
[3fa0cd96] + REPL
[9a3f8284] + Random
[ea8e919c] + SHA
[9e88b42a] + Serialization
[6462fe0b] + Sockets
[cf7118a7] + UUIDs
[4ec0a83e] + Unicode
┌ Error: build.log for PackageCompiler [9b87118b-4619-50d2-8e1e-99f35a4d4d9d] not found
└ @ Main /data/home/x/miniconda3/envs/foo/lib/python3.7/site-packages/julia/install-packagecompiler.jl:18
[ Info: Loading PackageCompiler...
[ Info: PackageCompiler is successfully installed at /tmp/tmp.pyjulia.sysimage.f6s6rkax/compiler_env
Error processing line 1 of /data/home/x/miniconda3/envs/foo/lib/python3.7/site-packages/matplotlib-3.1.3-py3.7-nspkg.pth:
Fatal Python error: initsite: Failed to import the site module
Traceback (most recent call last):
File "/data/home/x/miniconda3/envs/foo/lib/python3.7/site.py", line 168, in addpackage
exec(line)
File "<string>", line 1, in <module>
File "/data/home/x/miniconda3/envs/foo/lib/python3.7/importlib/util.py", line 14, in <module>
from contextlib import contextmanager
File "/data/home/x/miniconda3/envs/foo/lib/python3.7/contextlib.py", line 5, in <module>
from collections import deque
File "/data/home/x/miniconda3/envs/foo/lib/python3.7/collections/__init__.py", line 24, in <module>
import heapq as _heapq
File "/data/home/x/miniconda3/envs/foo/lib/python3.7/heapq.py", line 587, in <module>
from _heapq import *
SystemError: initialization of _heapq did not return an extension module
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/data/home/x/miniconda3/envs/foo/lib/python3.7/site.py", line 579, in <module>
main()
File "/data/home/x/miniconda3/envs/foo/lib/python3.7/site.py", line 566, in main
known_paths = addsitepackages(known_paths)
File "/data/home/x/miniconda3/envs/foo/lib/python3.7/site.py", line 349, in addsitepackages
addsitedir(sitedir, known_paths)
File "/data/home/x/miniconda3/envs/foo/lib/python3.7/site.py", line 207, in addsitedir
addpackage(sitedir, name, known_paths)
File "/data/home/x/miniconda3/envs/foo/lib/python3.7/site.py", line 178, in addpackage
import traceback
File "/data/home/x/miniconda3/envs/foo/lib/python3.7/traceback.py", line 3, in <module>
import collections
File "/data/home/x/miniconda3/envs/foo/lib/python3.7/collections/__init__.py", line 24, in <module>
import heapq as _heapq
File "/data/home/x/miniconda3/envs/foo/lib/python3.7/heapq.py", line 587, in <module>
from _heapq import *
SystemError: initialization of _heapq did not return an extension module
Command '['/home/x/miniconda3/envs/foo/bin/julia-py', '--julia', 'julia', '--color=yes', '/data/home/x/miniconda3/envs/foo/lib/python3.7/site-packages/julia/compile.jl', '/tmp/tmp.pyjulia.sysimage.f6s6rkax/compiler_env', '/data/home/x/miniconda3/envs/foo/lib/python3.7/site-packages/julia/precompile.jl', '/data/home/x/foo/code/sys.so']' returned non-zero exit status 1.
I'm trying to take the system image approach, but I'm getting the errors above. Any help please?
Hmm... maybe incompatible pre-compilation cache is used? IIRC rm -rf ~/.julia/compiled/v1.5/PyCall/ sometimes helps.
Thanks, I saw that on another thread and did delete it, and it made no difference.
On Wed, 15 Jul 2020, 02:34 Takafumi Arakaki, [email protected] wrote:
Hmm... maybe incompatible pre-compilation cache is used? IIRC rm -rf ~/.julia/compiled/v1.5/PyCall/ sometimes helps.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/JuliaPy/pyjulia/issues/389#issuecomment-658574705, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABA2BXL273LHQUC3DIUN6OTR3VEV3ANCNFSM4OZRCGIQ .
I did a bit more digging into this (amongst other things switched down to 1.4, which didn't make a difference)
Turns out there was some PyCall related stuff in my startup.jl that was intefering somehow. With that removed, if I run:
rm -rf ~/.julia/compiled/v1.4/PyCall
python3 -m julia.sysimage sys.so
I get a system image that I can use from pyjulia. Great!
What I'd actually like to do is have a system image with a bit more stuff in it. I have a script I use to build a useful system image with many of the packages I use using PackageCompiler, for my main project. If I try to use the system image generated with that in pyjulia, I get the same error about heapq.
I've slowly stripped that script away until it doesn't do much more than the PackgeCompiler script in pyjulia. Now it only compiles the PyCall package, calls a single precompile_execution_file which contians using PyCall, and writes out the sysimage. The only difference is that it uses my main project environment for all of this, rather than the isolated environment that PyJulia makes. This seems to be making the difference in terms of the above errors. Does that help you debug this?
A sysimage for PyJulia has to be compiled from a specific entry point. If you have a script.jl that contains the call to create_sysimage etc., maybe you can use julia-py script.jl or python3 -m julia.julia_py script.jl to create a system image.
That seems to have built correctly - thanks for all your help!
When I try to load it up, I'm getting this error now:
>>> import julia
>>> jl = julia.Julia(sysimage="/home/x/sysimage-py.so")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/x/miniconda3/envs/myenv/lib/python3.7/site-packages/julia/core.py", line 678, in __init__
self.__julia = Julia(*args, **kwargs)
File "/home/x/miniconda3/envs/myenv/lib/python3.7/site-packages/julia/core.py", line 482, in __init__
raise UnsupportedPythonError(jlinfo)
julia.core.UnsupportedPythonError: It seems your Julia and PyJulia setup are not supported.
Julia executable:
julia
Python interpreter and libpython used by PyCall.jl:
/home/x/miniconda3/envs/myenv/bin/python
/data/home/x/miniconda3/envs/myenv/lib/libpython3.7m.so.1.0
Python interpreter used to import PyJulia and its libpython.
/home/x/miniconda3/envs/myenv/bin/python
/data/home/x/miniconda3/envs/myenv/lib/libpython3.7m.so.1.0
Your Python interpreter "/home/x/miniconda3/envs/myenv/bin/python"
is statically linked to libpython. Currently, PyJulia does not fully
support such Python interpreter.
The easiest workaround is to pass `compiled_modules=False` to `Julia`
constructor. To do so, first *reboot* your Python REPL (if this happened
inside an interactive session) and then evaluate:
>>> from julia.api import Julia
>>> jl = Julia(compiled_modules=False)
Another workaround is to run your Python script with `python-jl`
command bundled in PyJulia. You can simply do:
$ python-jl PATH/TO/YOUR/SCRIPT.py
See `python-jl --help` for more information.
For more information, see:
https://pyjulia.readthedocs.io/en/latest/troubleshooting.html
It seems I can work around it with this:
from julia.api import LibJulia
api = LibJulia.load()
api.sysimage = "/home/x/sysimage-py.so"
api.init_julia() # works
Is that expected?
It also seems like the sysimage doesn't work with normal julia (segfaults), so I suppose I will have to build a separate sysimage for julia and for pyjulia?
Finally, is there a nicer way to inject the sysimage path so I don't have to set up LibJulia everytime? It would be cool if I could pass it as an environment variable, so my python code can just do from julia import X without requiring the setup to happen before the first import statement.
Is that expected?
Yeah, that's https://github.com/JuliaPy/pyjulia/issues/310#issuecomment-514034208 (still not fixed yet...)
I will have to build a separate sysimage for julia and for pyjulia?
After #380 (not released yet), you can create a custom system image for julia itself and then pass it to --base-sysimage to turn it to be PyJulia-compatible sysimage.
Finally, is there a nicer way to inject the sysimage path so I don't have to set up LibJulia everytime?
I want to implement a standard format for per-project system image https://github.com/JuliaLang/julia/pull/35794. Once that's in julia itself, we can implement a similar approach in PyJulia.
I gave the --base-sysimage ... argument a go, on 1.5.0. It seemed to crash as follows:
┌ Error: Failed to monkey-patch `julia` │ exception =
│ Evaluation into the closed module `Base` breaks incremental compilation because the side effects will not be permanent. This is likely due to some other module mutating `Base` with `eval` during precompilation - don't do this. │ Stacktrace:
│ [1] eval at ./boot.jl:331 [inlined]
│ [2] eval(::Module, ::Expr) at ./Base.jl:39
│ [3] top-level scope at /data/home/.../lib/python3.7/site-packages/julia/patch.jl:32
│ [4] include(::Function, ::Module, ::String) at ./Base.jl:380
│ [5] include(::Module, ::String) at ./Base.jl:368
│ [6] top-level scope at none:1
│ getpid() = 26723
└ @ Main /data/home/.../lib/python3.7/site-packages/julia/patch.jl:59
julia-py: Error in /data/home/.../lib/python3.7/site-packages/julia/patch.jl
ERROR: LoadError: Failed to precompile PyCall [438e738f-606a-5dbb-bf0a-cddfbfd45ab0] to /home/.../.julia/compiled/v1.5/PyCa
ll/tJuEG_OlfAW.ji.
Is this failing because my base sysimage includes PyCall already?
I think I solved this in #394. It's included in the latest released (PyJulia v0.5.4). What version did you try?
Is this failing because my base sysimage includes PyCall already?
I don't think the error itself is related to PyCall. But I don't think julia.sysimage itself would work when PyCall is already in the input sysimage.
Thanks. So to efficiently build two system images, one for julia use and one for pyjulia use, the best thing would be to build a heavy base image without PyCall, then build 2 incremental images on top of that?