conda-store
conda-store copied to clipboard
Parallel builds of environments
A conda environment install can be broken into several steps:
- solve environment from environment.yaml
- download and extract packages to package cache
- install conda environment
Currently all of this is done in https://github.com/Quansight/conda-store/blob/main/conda-store-server/conda_store_server/build.py#L97-L110 via conda env create -f environment.yaml -p <prefix>
in one monolithic approach. As a first approach (which can be improved later) we need to break this function into several steps.
Replace conda env create -f environment.yaml -p <prefix>
with
-
conda-lock -f environment.yml -p linux-64
this will generate a yaml file for the lock - Python code to inspect the conda lock yaml file (example here) and download each package with
manager: conda
to the conda package cache directory and extract. This step must use locking using filelock https://github.com/Quansight/conda-store/blob/main/conda-store-server/conda_store_server/build.py#L135-L137. -
conda-lock install -p <prefix>
You can get the package cache directory via the following. This will help determine where packages are downloaded an extracted.
from conda.base.context import context
context.pkgs_dirs
There are two formats that I know of for packages that are downloaded and need to be handled.
- .tar.gz (straightforward to handle)
- .conda https://docs.conda.io/projects/conda/en/latest/user-guide/concepts/packages.html#conda-file-format
@aktech was there any progress in implementing this? Happy to meet and discuss further if this is still a priority.
Closing as implemented in #417
See notes form a matrix conversation https://matrix.to/#/!SgckMCYyQUhdyooKZe:gitter.im/$U5CYJdRgHFBF6pqWSmP_h2HbY8mHTTl7wX-F1Po1JkI?via=gitter.im&via=matrix.org&via=tum.de