conda-lock
conda-lock copied to clipboard
Published
20 hours ago •
conda
conda
pip integration: conda_lock._vendor.poetry.utils._compat.CalledProcessError: Command ... returned non-zero exit status 128
Checklist
- [X] I added a descriptive title
- [X] I searched open reports and couldn't find a duplicate
What happened?
If you try to conda-lock the following environment:
environment.yml
channels:
- defaults
- pytorch
- nvidia
- conda-forge
dependencies:
- python=3.10
- pip
- pytorch=2.0.1=py3.10_cuda11.8_cudnn8.7.0_0
- torchvision=0.15.2=py310_cu118
- pytorch-cuda=11.8
- pip:
- git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
conda-lock --mamba -p linux-64
You will receive the following error:
❯ conda-lock --mamba -p linux-64
/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/pydantic/_internal/_config.py:269: UserWarning: Valid config keys have changed in V2:
* 'allow_mutation' has been removed
warnings.warn(message, UserWarning)
Locking dependencies for ['linux-64']...
INFO:conda_lock.conda_solver:linux-64 using specs ['python 3.10.*', 'pip *', 'pytorch 2.0.1 py3.10_cuda11.8_cudnn8.7.0_0', 'torchvision 0.15.2 py310_cu118', 'pytorch-cuda 11.8.*']
Traceback (most recent call last):
File "/home/michael/.local/bin/conda-lock", line 8, in <module>
sys.exit(main())
File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 1332, in lock
lock_func(
File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 1040, in run_lock
make_lock_files(
File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 381, in make_lock_files
lock_content = lock_content | create_lockfile_from_spec(
File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 787, in create_lockfile_from_spec
deps = _solve_for_arch(
File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 721, in _solve_for_arch
pip_deps = solve_pypi(
File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/conda_lock/pypi_solver.py", line 335, in solve_pypi
result = s.solve(use_latest=to_update)
File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/conda_lock/_vendor/poetry/puzzle/solver.py", line 65, in solve
packages, depths = self._solve(use_latest=use_latest)
File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/conda_lock/_vendor/poetry/puzzle/solver.py", line 233, in _solve
result = resolve_version(
File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/conda_lock/_vendor/poetry/mixology/__init__.py", line 7, in resolve_version
return solver.solve()
File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/conda_lock/_vendor/poetry/mixology/version_solver.py", line 84, in solve
next = self._choose_package_version()
File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/conda_lock/_vendor/poetry/mixology/version_solver.py", line 400, in _choose_package_version
version = self._provider.complete_package(version)
File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/conda_lock/_vendor/poetry/puzzle/provider.py", line 452, in complete_package
self.search_for_vcs(r)
File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/conda_lock/_vendor/poetry/puzzle/provider.py", line 163, in search_for_vcs
package = self.get_package_from_vcs(
File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/conda_lock/_vendor/poetry/puzzle/provider.py", line 193, in get_package_from_vcs
git.clone(url, tmp_dir)
File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/conda_lock/_vendor/poetry/core/vcs/git.py", line 262, in clone
return self.run("clone", "--recurse-submodules", "--", repository, str(dest))
File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/conda_lock/_vendor/poetry/core/vcs/git.py", line 356, in run
subprocess.check_output(
File "/usr/lib/python3.10/subprocess.py", line 421, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/conda_lock/_vendor/poetry/utils/_compat.py", line 217, in run
raise CalledProcessError(
conda_lock._vendor.poetry.utils._compat.CalledProcessError: Command '['git', 'clone', '--recurse-submodules', '--', 'https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch', '/tmp/pypoetry-git-torchc39fhmwj']' returned non-zero exit status 128.
Conda Info
mamba 1.4.2
conda 23.3.1
Conda Config
❯ conda config --show-sources
==> /home/michael/mambaforge/.condarc <==
channels:
- conda-forge
Conda list
❯ conda list --show-channel-urls
# packages in environment at /home/michael/mambaforge:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
boltons 23.0.0 pyhd8ed1ab_0 conda-forge
brotli-python 1.0.9 py310hd8f1fbe_9 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
c-ares 1.19.1 hd590300_0 conda-forge
ca-certificates 2023.7.22 hbcca054_0 conda-forge
certifi 2023.7.22 pyhd8ed1ab_0 conda-forge
cffi 1.15.1 py310h255011f_3 conda-forge
charset-normalizer 3.2.0 pyhd8ed1ab_0 conda-forge
colorama 0.4.6 pyhd8ed1ab_0 conda-forge
conda 23.3.1 py310hff52083_0 conda-forge
conda-libmamba-solver 23.3.0 pyhd8ed1ab_0 conda-forge
conda-package-handling 2.2.0 pyh38be061_0 conda-forge
conda-package-streaming 0.9.0 pyhd8ed1ab_0 conda-forge
cryptography 41.0.3 py310h75e40e8_0 conda-forge
fmt 9.1.0 h924138e_0 conda-forge
icu 72.1 hcb278e6_0 conda-forge
idna 3.4 pyhd8ed1ab_0 conda-forge
jsonpatch 1.32 pyhd8ed1ab_0 conda-forge
jsonpointer 2.0 py_0 conda-forge
keyutils 1.6.1 h166bdaf_0 conda-forge
krb5 1.21.2 h659d440_0 conda-forge
ld_impl_linux-64 2.40 h41732ed_0 conda-forge
libarchive 3.6.2 h039dbb9_1 conda-forge
libcurl 8.2.1 hca28451_0 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 h516909a_1 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc-ng 13.1.0 he5830b7_0 conda-forge
libgomp 13.1.0 he5830b7_0 conda-forge
libiconv 1.17 h166bdaf_0 conda-forge
libmamba 1.4.2 hcea66bb_0 conda-forge
libmambapy 1.4.2 py310h1428755_0 conda-forge
libnghttp2 1.52.0 h61bc06f_0 conda-forge
libnsl 2.0.0 h7f98852_0 conda-forge
libsolv 0.7.24 hfc55251_1 conda-forge
libsqlite 3.42.0 h2797004_0 conda-forge
libssh2 1.11.0 h0841786_0 conda-forge
libstdcxx-ng 13.1.0 hfd8a6a1_0 conda-forge
libuuid 2.38.1 h0b41bf4_0 conda-forge
libxml2 2.11.5 h0d562d8_0 conda-forge
libzlib 1.2.13 hd590300_5 conda-forge
lz4-c 1.9.4 hcb278e6_0 conda-forge
lzo 2.10 h516909a_1000 conda-forge
mamba 1.4.2 py310h51d5547_0 conda-forge
ncurses 6.4 hcb278e6_0 conda-forge
openssl 3.1.2 hd590300_0 conda-forge
packaging 23.1 pyhd8ed1ab_0 conda-forge
pip 23.2.1 pyhd8ed1ab_0 conda-forge
pluggy 1.2.0 pyhd8ed1ab_0 conda-forge
pybind11-abi 4 hd8ed1ab_3 conda-forge
pycosat 0.6.4 py310h5764c6d_1 conda-forge
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pyopenssl 23.2.0 pyhd8ed1ab_1 conda-forge
pysocks 1.7.1 pyha2e5f31_6 conda-forge
python 3.10.12 hd12c33a_0_cpython conda-forge
python_abi 3.10 3_cp310 conda-forge
readline 8.2 h8228510_1 conda-forge
reproc 14.2.4 h0b41bf4_0 conda-forge
reproc-cpp 14.2.4 hcb278e6_0 conda-forge
requests 2.31.0 pyhd8ed1ab_0 conda-forge
ruamel.yaml 0.17.32 py310h2372a71_0 conda-forge
ruamel.yaml.clib 0.2.7 py310h1fa729e_1 conda-forge
setuptools 68.1.2 pyhd8ed1ab_0 conda-forge
tk 8.6.12 h27826a3_0 conda-forge
toolz 0.12.0 pyhd8ed1ab_0 conda-forge
tqdm 4.66.1 pyhd8ed1ab_0 conda-forge
tzdata 2023c h71feb2d_0 conda-forge
urllib3 2.0.4 pyhd8ed1ab_0 conda-forge
wheel 0.41.1 pyhd8ed1ab_0 conda-forge
xz 5.2.6 h166bdaf_0 conda-forge
yaml-cpp 0.7.0 h27087fc_2 conda-forge
zstandard 0.19.0 py310h1275a96_2 conda-forge
zstd 1.5.2 hfc55251_7 conda-forge
### Additional Context
_No response_
Also happens with other git repos, e.g. git+https://github.com/luca-medeiros/lang-segment-anything.git
Of note, this and https://github.com/conda/conda-lock/issues/395 are the only blockers for using conda-lock for my project.
@maresb This is an old issue so hoping to bring it up, I ran into the same issue. It seems that when parsing environment.yml, the # is not handled correctly and gets treated as part of the repository name.
Thanks @anuraaga for the info, and I'm sorry you're having problems with conda-lock.
It seems that when parsing
environment.yml, the#is not handled correctly and gets treated as part of the repository name.
Would you be able to explain in more detail exactly what you mean by this comment? Thanks!
Ah sorry it wasn't clear. PIP supports parameters for vcs dependencies past #
https://pip.pypa.io/en/stable/cli/pip_install/#vcs-support
For example
python -m pip install -e 'git+https://git.repo/some_repo.git#egg=subdir&subdirectory=subdir_path' # install a python package from a repo subdirectory
But we are from the error message that it seems git clone is being executed with the contents of # which should not be part of the clone command, the final line of the original post
File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/conda_lock/_vendor/poetry/utils/_compat.py", line 217, in run
raise CalledProcessError(
conda_lock._vendor.poetry.utils._compat.CalledProcessError: Command '['git', 'clone', '--recurse-submodules', '--', 'https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch', '/tmp/pypoetry-git-torchc39fhmwj']' returned non-zero exit status 128
I can confirm having similar behavior when trying to use #subdirectory with a similar dependency
Thanks a lot @anuraaga for the detailed explanation!
The error seems to be occurring within the old version of Poetry that I vendored into conda-lock. I suspect that upgrading this vendored Poetry would resolve the issue. I have been meaning to tackle this for some time now. I hope to be able to finally get to it in early 2024, but there are no promises.