Conda.jl
Conda.jl copied to clipboard
JuliaPy
Fails to find Python 3.7 interpreter on NixOS 18.09, despite it being in PATH
Hi,
I've been trying to install IJulia using Julia's package manager, and Conda is a dependency of it that doesn't seem to want to cooperate with me. This is the error I'm getting:
┌ Error: Error building `IJulia`:
│ % Total % Received % Xferd Average Speed Time Time Time Current
│ Dload Upload Total Spent Left Speed
100 59.6M 100 59.6M 0 0 5207k 0 0:00:11 0:00:11 --:--:-- 5199k
│ PREFIX=/home/fusion809/.julia/conda/3
│ reinstalling: python-3.7.0-hc3d631a_0 ...
│ /home/fusion809/.julia/conda/3/installer.sh: line 370: /home/fusion809/.julia/conda/3/pkgs/python-3.7.0-hc3d631a_0/bin/python: No such file or directory
│ ┌ Warning: Could not execute `jupyter --version`.
│ └ @ Main ~/.julia/packages/IJulia/DL02A/deps/build.jl:41
│ [ Info: Installing Jupyter via the Conda package.
│ [ Info: Downloading miniconda installer ...
│ [ Info: Installing miniconda ...
│ ERROR: LoadError: failed process: Process(`/home/fusion809/.julia/conda/3/installer.sh -b -f -p /home/fusion809/.julia/conda/3`, ProcessExited(1)) [1]
│ Stacktrace:
/home/fusion809/.julia/conda/3/pkgs/python-3.7.0-hc3d631a_0/bin/python exists, it is symlink to python37, which is a command in PATH. Although, I should mention NixOS places binaries in non-FHS locations, and while my user PATH environment variable points to ~/.nix-profile/bin, where python3.7 is found, it is not in the usual FHS location of /usr/bin.
Thanks for your time, Brenton
Maybe it's helpful to see the output of
cd ~/.julia/conda/3/pkgs/python-3.7.0-hc3d631a_0/bin/
ls -lh python*
I happened to have the same package in the cache:
$ cd ~/miniconda3/pkgs/python-3.7.0-hc3d631a_0/bin
$ ls -lh python*
lrwxrwxrwx 1 takafumi takafumi 9 Jul 29 15:11 python -> python3.7
lrwxrwxrwx 1 takafumi takafumi 9 Jul 29 15:11 python3 -> python3.7
-rwxrwxr-x 1 takafumi takafumi 3.8M Jun 28 06:29 python3.7
lrwxrwxrwx 1 takafumi takafumi 17 Jul 29 15:11 python3.7-config -> python3.7m-config
lrwxrwxrwx 1 takafumi takafumi 9 Jul 29 15:11 python3.7m -> python3.7
-rwxrwxr-x 1 takafumi takafumi 3.5K Jun 28 06:29 python3.7m-config
lrwxrwxrwx 1 takafumi takafumi 17 Jul 29 15:11 python3-config -> python3.7m-config
You may also want to check if manually installing miniconda works. It's not clear if the issue is in Conda.jl.
lrwxrwxrwx 1 fusion809 users 9 Jun 28 23:29 python -> python3.7
lrwxrwxrwx 1 fusion809 users 9 Jun 28 23:29 python3 -> python3.7
-rwxr-xr-x 1 fusion809 users 3.8M Jun 28 23:29 python3.7
lrwxrwxrwx 1 fusion809 users 17 Jun 28 23:29 python3.7-config -> python3.7m-config
lrwxrwxrwx 1 fusion809 users 9 Jun 28 23:29 python3.7m -> python3.7
-rwxr-xr-x 1 fusion809 users 3.5K Jun 28 23:29 python3.7m-config
lrwxrwxrwx 1 fusion809 users 17 Jun 28 23:29 python3-config -> python3.7m-config
So python3.7 exists. I wonder why the installer script says it doesn't.
If you want to investigate more, here is the installation guide of miniconda: https://conda.io/docs/user-guide/install/linux.html
(My deleted comment was when I forgot I was on Arch, instead of NixOS) Turns out I get the same error with it:
./Miniconda3-latest-Linux-x86_64.sh: line 370: /home/fusion809/miniconda3/pkgs/python-3.7.0-hc3d631a_0/bin/python: No such file or directory
Thanks for reporting it to upstream!
I noticed nix has conda as a package https://github.com/NixOS/nixpkgs/issues/26245. Maybe you can install it this way and point environment variable CONDA_JL_HOME to it https://github.com/JuliaPy/Conda.jl#using-a-pre-existing-conda-installation
Thanks mate for trying to help me, although Python packages, especially Python package managers, on NixOS 18.09 have a bad habit of biting me in the rear end. Largely because using them the way you do on most Linux distros tends to return errors relating to the non-FHS compliance of NixOS. I cannot use i3pystatus on NixOS due to these issues, I've tried following their Wiki but pip and even their own i3pystatus package do not seem to want to play ball. The conda command itself throws this error on it: https://gist.github.com/fusion809/52f9d67f99aa96bc40780ca072a02550.
Hmm... That's too bad. What is bizarre is that the installation script says pkgs/python-3.7.0-hc3d631a_0/bin/python does not exists when it does. I also don't see why it would be related to non-FHS layout. Anyway, let's hope conda devs can help.
I think this is because miniconda expects libraries in /lib64 and possibly different versions. I get similar results:
b818679acc42:/# bash ~/miniconda.sh -b -p /opt/conda -s
PREFIX=/opt/conda
installing: python-3.7.0-hc3d631a_0 ...
/root/miniconda.sh: line 370: /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: No such file or directory
b818679acc42:/# ldd /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python
/lib64/ld-linux-x86-64.so.2 (0x7f6e26639000)
libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7f6e26639000)
libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f6e26639000)
libdl.so.2 => /lib64/ld-linux-x86-64.so.2 (0x7f6e26639000)
libutil.so.1 => /lib64/ld-linux-x86-64.so.2 (0x7f6e26639000)
librt.so.1 => /lib64/ld-linux-x86-64.so.2 (0x7f6e26639000)
libm.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f6e26639000)
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __snprintf_chk: symbol not found
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __wcscat_chk: symbol not found
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __isnan: symbol not found
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __finite: symbol not found
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __isinf: symbol not found
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __isinff: symbol not found
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __realpath_chk: symbol not found
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __memmove_chk: symbol not found
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __memcpy_chk: symbol not found
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __wcsncpy_chk: symbol not found
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __wcscpy_chk: symbol not found
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __rawmemchr: symbol not found
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __open64_2: symbol not found
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __sched_cpufree: symbol not found
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __sched_cpualloc: symbol not found
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __vsnprintf_chk: symbol not found
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __strncpy_chk: symbol not found
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __strcpy_chk: symbol not found
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __printf_chk: symbol not found
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __fread_chk: symbol not found
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __fprintf_chk: symbol not found
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __sprintf_chk: symbol not found
I think this is solved by people using buildFHSUserEnv, ie: https://github.com/NixOS/nixpkgs/blob/db558b31b9af9db59a8000c85c21057c88efcc25/pkgs/tools/package-management/conda/default.nix#L55
I don't understand why buildFHSUserEnv is required instead of being able to use patchelf to just repoint things, but that's what I'm trying to understand now.
Hopefully this helps.
Yeah but buildFHSUserEnv is only usable when you're building a package, not when you're using NixOS and trying to install Conda.jl with Julia.
I don't understand why buildFHSUserEnv is required instead of being able to use patchelf to just repoint things, but that's what I'm trying to understand now.
The conda installer extracts the Python ELF binary and expects to be able to run it immediately, if you want to insert patchelf, you need to modify the conda installer (and you'd also have to use patchelf on every conda package you install that contains ELFs - it gets difficult).
Any progress here? I'm experiencing the same problem with Julia 1.0.4 on Ubuntu 18.04 LTS and 19.04.
julia> Conda.list()
[ Info: Downloading miniconda installer ...
--2019-10-02 08:53:06-- https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86.sh
Resolving repo.continuum.io (repo.continuum.io)... 104.18.200.79, 104.18.201.79, 2606:4700::6812:c94f, ...
Connecting to repo.continuum.io (repo.continuum.io)|104.18.200.79|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 65741329 (63M) [application/x-sh]
Saving to: ‘/home/ubuntu/.julia/conda/3/installer.sh’
/home/ubuntu/.julia 100%[===================>] 62,70M 9,87MB/s in 6,4s
2019-10-02 08:53:12 (9,81 MB/s) - ‘/home/ubuntu/.julia/conda/3/installer.sh’ saved [65741329/65741329]
[ Info: Installing miniconda ...
PREFIX=/home/ubuntu/.julia/conda/3
reinstalling: python-3.7.1-h0371630_7 ...
/home/ubuntu/.julia/conda/3/installer.sh: 373: /home/ubuntu/.julia/conda/3/installer.sh: /home/ubuntu/.julia/conda/3/pkgs/python-3.7.1-h0371630_7/bin/python: not found
ERROR: failed process: Process(`/home/ubuntu/.julia/conda/3/installer.sh -b -f -p /home/ubuntu/.julia/conda/3`, ProcessExited(1)) [1]
Stacktrace:
[1] error(::String, ::Base.Process, ::String, ::Int64, ::String) at ./error.jl:42
[2] pipeline_error at ./process.jl:705 [inlined]
[3] #run#505(::Bool, ::Function, ::Cmd) at ./process.jl:663
[4] run at ./process.jl:661 [inlined]
[5] _install_conda(::String, ::Bool) at /home/ubuntu/.julia/packages/Conda/kLXeC/src/Conda.jl:165
[6] _install_conda(::String) at /home/ubuntu/.julia/packages/Conda/kLXeC/src/Conda.jl:152
[7] runconda(::Cmd, ::String) at /home/ubuntu/.julia/packages/Conda/kLXeC/src/Conda.jl:111
[8] list(::String) at /home/ubuntu/.julia/packages/Conda/kLXeC/src/Conda.jl:225 (repeats 2 times)
[9] top-level scope at none:0
Any progress here? I'm experiencing the same problem with Julia 1.0.4 on Ubuntu 18.04 LTS and 19.04.
I solved my issue. It turned out that I had a 32-bit Julia installed on a 64-bit Ubuntu, and Conda.jl tried to install a 32-bit miniconda which failed. Using a 64-bit Julia works.
Thanks @ahumenberger, I installed Julia via snap and was having this issue, but reinstallling the correct Julia binaries solved the problem.
