PythonCall.jl icon indicating copy to clipboard operation
PythonCall.jl copied to clipboard
verified publisher icon JuliaPy

python.exe not found

Open gyansinha opened this issue 2 years ago • 12 comments

IOError: could not spawn 'H:\.julia\environments\v1.9\.CondaPkg\env\python.exe' --version: no such file or directory (ENOENT)

It looks like the python executable isn't there at all after "add PythonCall" - any ideas why this may be happening.

gyansinha avatar Oct 12 '23 17:10 gyansinha

It'll be much easier to help if you give the full sequence of steps you did to get to that error, plus the full error message.

cjdoris avatar Oct 12 '23 21:10 cjdoris

My guess though is that your antivirus deleted python.exe after it was installed.

cjdoris avatar Oct 12 '23 21:10 cjdoris

It'll be much easier to help if you give the full sequence of steps you did to get to that error, plus the full error message.

add PythonCall using PythonCall add quantlib

] conda resolve

This is the error I get:

CondaPkg Creating environment
         │ H:\.julia\artifacts\4b87dc8b933ca2d289a392e264aaf44df2410f6f\bin\micromamba.exe
         │ -r H:\.julia\scratchspaces\0b3b1443-0f03-428d-bdfb-f27f9c1191ea\root
         │ create
         │ -y
         │ -p \\PROD-FILE\XXXX$\User\gsinha\.julia\dev\Clover\.CondaPkg\env
         │ --override-channels
         │ --no-channel-priority
         │ python[version='>=3.7,<4',channel='conda-forge',build='*cpython*']
         │ quantlib[version='*']
         └ -c conda-forge

critical libmamba remove: Access is denied.: "\PROD-FILE\XXXX$\User\gsinha.julia\dev\Clover.CondaPkg\env\DLLs_ctypes.pyd.mamba_trash" ERROR: failed process: Process('H:\.julia\artifacts\4b87dc8b933ca2d289a392e264aaf44df2410f6f\bin\micromamba.exe' -r 'H:\.julia\scratchspaces\0b3b1443-0f03-428d-bdfb-f27f9c1191ea\root' create -y -p '\\PROD-FILE\XXXX$\User\gsinha\.julia\dev\Clover\.CondaPkg\env' --override-channels --no-channel-priority "python[version='>=3.7,<4',channel='conda-forge',build='*cpython*']" "quantlib[version='*']" -c conda-forge, ProcessExited(1)) [1]

Stacktrace: [1] pipeline_error @ .\process.jl:565 [inlined] [2] run(::Cmd; wait::Bool) @ Base .\process.jl:480 [3] run(::Cmd) @ Base .\process.jl:477 [4] _run(io::IO, cmd::Cmd, args::Any; flags::Any) @ CondaPkg H:.julia\packages\CondaPkg\22jbl\src\resolve.jl:398 [5] _resolve_conda_install(io::Any, conda_env::Any, specs::Any, channels::Any; create::Any) @ CondaPkg H:.julia\packages\CondaPkg\22jbl\src\resolve.jl:299 [6] resolve(; force::Bool, io::IO, interactive::Bool, dry_run::Bool) @ CondaPkg H:.julia\packages\CondaPkg\22jbl\src\resolve.jl:547 [7] resolve(; force::Bool) @ CondaPkg.PkgREPL H:.julia\packages\CondaPkg\22jbl\src\PkgREPL.jl:82 [8] resolve() @ CondaPkg.PkgREPL H:.julia\packages\CondaPkg\22jbl\src\PkgREPL.jl:81 [9] do_cmd!(command::Pkg.REPLMode.Command, repl::REPL.LineEditREPL) @ Pkg.REPLMode C:\Program Files\Julia-1.9.2\share\julia\stdlib\v1.9\Pkg\src\REPLMode\REPLMode.jl:409 [10] do_cmd(repl::REPL.LineEditREPL, input::String; do_rethrow::Bool) @ Pkg.REPLMode C:\Program Files\Julia-1.9.2\share\julia\stdlib\v1.9\Pkg\src\REPLMode\REPLMode.jl:390 [11] do_cmd @ C:\Program Files\Julia-1.9.2\share\julia\stdlib\v1.9\Pkg\src\REPLMode\REPLMode.jl:380 [inlined] [12] (::Pkg.REPLMode.var"#24#27"{REPL.LineEditREPL, REPL.LineEdit.Prompt})(s::REPL.LineEdit.MIState, buf::IOBuffer, ok::Bool) @ Pkg.REPLMode C:\Program Files\Julia-1.9.2\share\julia\stdlib\v1.9\Pkg\src\REPLMode\REPLMode.jl:557 [13] #invokelatest#2 @ .\essentials.jl:816 [inlined] [14] invokelatest @ .\essentials.jl:813 [inlined] [15] run_interface(terminal::REPL.Terminals.TextTerminal, m::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState) @ REPL.LineEdit C:\Program Files\Julia-1.9.2\share\julia\stdlib\v1.9\REPL\src\LineEdit.jl:2647 [16] run_frontend(repl::REPL.LineEditREPL, backend::REPL.REPLBackendRef) @ REPL C:\Program Files\Julia-1.9.2\share\julia\stdlib\v1.9\REPL\src\REPL.jl:1300 [17] (::REPL.var"#62#68"{REPL.LineEditREPL, REPL.REPLBackendRef})() @ REPL .\task.jl:514

I am having the anti-virus looked into - thanks for your tip on that.

gyansinha avatar Oct 13 '23 14:10 gyansinha

After this step, I do the followiing after exiting Julia entirely:

julia> using PythonCall ERROR: InitError: Python executable "h:\.julia\dev\Clover\.CondaPkg\env\python.exe" is not executable. Stacktrace: [1] error(s::String) @ Base .\error.jl:35 [2] init_context() @ PythonCall.C H:.julia\packages\PythonCall\qTEA1\src\cpython\context.jl:88 [3] init() @ PythonCall.C H:.julia\packages\PythonCall\qTEA1\src\cpython\CPython.jl:21 [4] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String) @ Base .\loading.jl:1115 [5] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any}) @ Base .\loading.jl:1061 [6] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128) @ Base .\loading.jl:1506 [7] _require(pkg::Base.PkgId, env::String) @ Base .\loading.jl:1783 [8] _require_prelocked(uuidkey::Base.PkgId, env::String) @ Base .\loading.jl:1660 [9] macro expansion @ .\loading.jl:1648 [inlined] [10] macro expansion @ .\lock.jl:267 [inlined] [11] require(into::Module, mod::Symbol) @ Base .\loading.jl:1611 during initialization of module C

caused by: IOError: could not spawn 'h:\.julia\dev\Clover\.CondaPkg\env\python.exe' --version: no such file or directory (ENOENT) Stacktrace: [1] _spawn_primitive(file::String, cmd::Cmd, stdio::Vector{Union{RawFD, Base.Libc.WindowsRawSocket, IO}}) @ Base .\process.jl:128 [2] #760 @ .\process.jl:139 [inlined] [3] setup_stdios(f::Base.var"#760#761"{Cmd}, stdios::Vector{Union{RawFD, Base.Libc.WindowsRawSocket, IO}}) @ Base .\process.jl:223 [4] _spawn @ .\process.jl:138 [inlined] [5] _spawn(::Base.CmdRedirect, ::Vector{Union{RawFD, Base.Libc.WindowsRawSocket, IO}}) (repeats 2 times) @ Base .\process.jl:166 [6] run(::Base.CmdRedirect; wait::Bool) @ Base .\process.jl:479 [7] run @ .\process.jl:477 [inlined] [8] init_context() @ PythonCall.C H:.julia\packages\PythonCall\qTEA1\src\cpython\context.jl:86 [9] init() @ PythonCall.C H:.julia\packages\PythonCall\qTEA1\src\cpython\CPython.jl:21 [10] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String) @ Base .\loading.jl:1115 [11] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any}) @ Base .\loading.jl:1061 [12] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128) @ Base .\loading.jl:1506 [13] _require(pkg::Base.PkgId, env::String) @ Base .\loading.jl:1783 [14] _require_prelocked(uuidkey::Base.PkgId, env::String) @ Base .\loading.jl:1660 [15] macro expansion @ .\loading.jl:1648 [inlined] [16] macro expansion @ .\lock.jl:267 [inlined] [17] require(into::Module, mod::Symbol) @ Base .\loading.jl:1611

gyansinha avatar Oct 13 '23 14:10 gyansinha

Can you remove that .CondaPkg folder entirely and try again?

cjdoris avatar Oct 17 '23 17:10 cjdoris

I am running into a similar behaviour by first installing some packages with CondaPkg, before even installing PythonCall.

It seems to me that this order is not supported currently. A workaround is to first install PythonCall into the environment before the first time import CondaPkg is executed in that environment. Then the __init__ of CondaPkg will automatically pickup all dependencies from PythonCall.

Unfortunately that workaround does not work for me, because my package depends on RCall, which I like to build on top of CondaPkg. However unlike PythonCall, RCall requires a ready R installation before it can be installed, hence I need to import CondaPkg before installing my package which depends on R. Hence all the Conda dependencies from my package are neither picked up nor resolved by CondaPkg.

So for me the only workaround left is to

  • delete .CondaPkg after RCall is installed, using rm(dirname(CondaPkg.envdir()), recursive=true)
  • and run import CondaPkg; CondaPkg.resolve() again after I added my package.

schlichtanders avatar Nov 06 '23 20:11 schlichtanders

@schlichtanders That order should definitely work.

Can you give the sequence of steps to reproduce your error? And also give the full error you see.

cjdoris avatar Nov 06 '23 21:11 cjdoris

It was similar to

julia -e 'import Pkg; \
    Pkg.Registry.add("General"); \
    Pkg.add("CondaPkg"); \
    import CondaPkg; \
    CondaPkg.add("r-base"); \
    ENV["R_HOME"] = CondaPkg.envdir() * "/lib/R"; \
    Pkg.add("RCall"); \
    Pkg.add("PythonCall"); \
    Pkg.precompile(); \
'

schlichtanders avatar Nov 08 '23 12:11 schlichtanders

Here a complete reproducible example

  1. Start a docker container with docker run -it --rm julia:1.9-bookworm bash
  2. run the above installation instructions
julia -e 'import Pkg; \
    Pkg.Registry.add("General"); \
    Pkg.add("CondaPkg"); \
    import CondaPkg; \
    CondaPkg.add("r-base"); \
    ENV["R_HOME"] = CondaPkg.envdir() * "/lib/R"; \
    Pkg.add("RCall"); \
    Pkg.add("PythonCall"); \
    Pkg.precompile(); \
'
  1. run julia -e "import PythonCall"

then the output is

ERROR: InitError: Python executable "/root/.julia/environments/v1.9/.CondaPkg/env/bin/python" is not executable.
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] init_context()
    @ PythonCall.C ~/.julia/packages/PythonCall/wXfah/src/cpython/context.jl:88
  [3] __init__()
    @ PythonCall.C ~/.julia/packages/PythonCall/wXfah/src/cpython/CPython.jl:21
  [4] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)
    @ Base ./loading.jl:1115
  [5] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any})
    @ Base ./loading.jl:1061
  [6] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128)
    @ Base ./loading.jl:1506
  [7] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1783
  [8] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1660
  [9] macro expansion
    @ ./loading.jl:1648 [inlined]
 [10] macro expansion
    @ ./lock.jl:267 [inlined]
 [11] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1611
during initialization of module C

caused by: IOError: could not spawn `/root/.julia/environments/v1.9/.CondaPkg/env/bin/python --version`: no such file or directory (ENOENT)
Stacktrace:
  [1] _spawn_primitive(file::String, cmd::Cmd, stdio::Vector{Union{RawFD, IO}})
    @ Base ./process.jl:128
  [2] #760
    @ ./process.jl:139 [inlined]
  [3] setup_stdios(f::Base.var"#760#761"{Cmd}, stdios::Vector{Union{RawFD, IO}})
    @ Base ./process.jl:223
  [4] _spawn
    @ ./process.jl:138 [inlined]
  [5] _spawn(::Base.CmdRedirect, ::Vector{Union{RawFD, IO}}) (repeats 2 times)
    @ Base ./process.jl:166
  [6] run(::Base.CmdRedirect; wait::Bool)
    @ Base ./process.jl:479
  [7] run
    @ ./process.jl:477 [inlined]
  [8] init_context()
    @ PythonCall.C ~/.julia/packages/PythonCall/wXfah/src/cpython/context.jl:86
  [9] __init__()
    @ PythonCall.C ~/.julia/packages/PythonCall/wXfah/src/cpython/CPython.jl:21
 [10] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)
    @ Base ./loading.jl:1115
 [11] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any})
    @ Base ./loading.jl:1061
 [12] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128)
    @ Base ./loading.jl:1506
 [13] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1783
 [14] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1660
 [15] macro expansion
    @ ./loading.jl:1648 [inlined]
 [16] macro expansion
    @ ./lock.jl:267 [inlined]
 [17] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1611

Note that this is only a dummy example. Instead of PythonCall the package which is actually needed is MyPackage.

  • MyPackage depends on RCall AND CondaPkg, defining a couple of R dependencies via CondaPkg.toml.
  • Hence MyPackage is a bit like PythonCall in this example, in that it depends on CondaPkg.
  • And a bit like RCall in that adding/importing MyPackage would build RCall itself, but RCall's R_HOME should be set by CondaPkg.envdir(). Hence MyPackage needs to be added after CondaPkg.resolve()/CondaPkg.envdir().

The ideal solution for me would be that RCall is resolving its dependency dynamically, but as of now it is not foreseeable if this will ever happen.

It would be great if CondaPkg can resolve newly added other julia dependencies which also include CondaPkg dependencies.

schlichtanders avatar Nov 22 '23 21:11 schlichtanders

Ok I see, so the issue is that you (implicitly) resolve the Conda environment in order to build RCall and then load some other packages, which also have Conda dependencies, and these aren't installed because CondaPkg doesn't auto resolve this time.

The reason for this is that CondaPkg doesn't resolve if you've already resolved, unless you explicitly add any packages.

A nice solution that I think you were getting at is to also resolve if the installed Julia packages change.

Is it possible to cheaply detect if the Julia environment has changed? Maybe Pkg tracks the time that the last resolve happened? Or we could see if Pkg.project() changed but that might be quite a slow check.

cjdoris avatar Nov 29 '23 19:11 cjdoris

Ok after a discussion on Slack the best approach seems to be to check if the mtime/hash of Manifest.toml changed since the last resolve.

cjdoris avatar Nov 30 '23 17:11 cjdoris

Sounds great! Thank you for inspecting/discussing this further

schlichtanders avatar Nov 30 '23 21:11 schlichtanders