Conda.jl
Conda.jl copied to clipboard
JuliaPy
Using Mamba instead of Conda
mamba is a dropin replacement for the conda package manager, which uses the same archive format as conda. Being written in c++, it is much much faster than conda itself, which matters a lot to me given the number of time, I happen to be recreating Julia/Python environement. Given it takes the same options as conda, I suppose it could be swapped or used optionally in Conda.jl with relatively little effort.
There is also an alternative, micromamba, which might also be useful in the Julia context, since it can be used to bootstrap a conda environment with no dependence on a Python executable (for instance, to install R libraries).
It might be possible to support this as an install-time option, but I'm reluctant to make it the default lest we introduce some incompatibilities.
(Hopefully, once Mamba fully implements conda functionality and passes the conda test-suite, Anaconda itself will adopt it if it is so much faster. I'm reluctant to get ahead of Anaconda itself here.)
I understand your point about introducing incompatibilities.
Having an option to do that would be great. I suppose those who opt-in would end-up with both conda and mamba installed in their environment so that it is possible to use mamba for installation (where it really shines, and possibly passes the appropriate tests?) and conda when the functionalities are missing.
We recently published this blog post about mamba, and we actually included some comments on how it may be really amenable to Julia bindings.
Mamba is very actively developed, on funded time, by several people. It has been adopted at scale by conda-forge for large software migrations, and is also in use in the bioconda community. We test mamba continuously, and dogfoot it in our team - but there isn't a "golden test suite" from conda that it would make sense to run on mamba
Happy to help with any question!
A first step might be to package micromamba or similar on Yggdrassil — this is the most convenient way to handle binary dependencies in Julia. (Basically, we write a build script, and then Yggdrasil automatically cross-compiles binaries for all supported platforms.)
It is possible to call Mamba as a library? Ideally with C bindings, as those are the easiest to access cross-language? (It would be great to avoid spawning a process for every command.)
It would be amazing if a Mamba.jl package replacing Conda with Mamba was available. Conda is terribly slow. Any progress on this since @SylvainCorlay? Many thanks in advance!
Someone needs to create a Yggdrasil build script, to start with, for us to easily use this in Julia.
You can already use mamba with Conda.jl.
julia> using Conda
julia> Conda.add("mamba")
julia> ENV["CONDA_JL_CONDA_EXE"] = joinpath(Conda.ROOTENV, "bin", "mamba")
pkg> build Conda
and then restart julia.
If it worked, you should be able to see something like below
julia> Conda.CONDA_EXE
"/home/isuru/.julia/conda/3/bin/mamba"
For
windows,joinpath(Conda.ROOTENV, "bin", "mamba")would change slightly.
Yip, it would be
ENV["CONDA_JL_CONDA_EXE"] = joinpath(Conda.ROOTENV, "condabin", "mamba.bat")
Just tried and worked like a charm. Much shorter installation times 😄
EDIT: Not sure whether it should be joinpath(Conda.ROOTENV, "Scripts", "mamba.exe") instead ...
I have started managing my python packages outside of julia with mamba. I think, then, mamba is managing environment of the shell from which I launch julia or python.
Is it possible that this outside mamba is interfering with the default conda installation used by julia?
I have hangs and crashes when, e.g., using PyPlot or pyimport("numpy"). I think it's possible that this is correlated to switching to mamba.
Do I need to reset some environment variables when building Conda (or PyCall) to make this work? I already do ENV["PYTHON"]="".
I was trying things... Basically, trying to run Julia inside Python using micromamba for python deps management. When trying to install PyCall, it fails, because it uses --satisfied-skip-solve e.g. this issue. Not sure if it should be solved there or here. I think here is a better option, e.g. add a check here.
