poetry-plugin-export
poetry-plugin-export copied to clipboard
python-poetry
Dependency walk failed
- Poetry version: 1.3.2
- Python version: 3.8
- OS version and name: wsl ubuntu 20.04
- pyproject.toml: https://gist.github.com/SunnyR/eedd0f0d974b4c046a660f354eb65dee
- [x] I am on the latest stable Poetry version, installed using a recommended method.
- [x] I have searched the issues of this repo and believe that this is not a duplicate.
- [x] I have consulted the FAQ and blog for any relevant entries or release notes.
- [x] If an exception occurs when executing a command, I executed it again in debug mode (
-vvvoption) and have included the output below.
Issue
When using poetry export with dev dependencies (following command), i get stack trace below with error:
poetry export -f requirements.txt -o ./dev_dependencies --without-hashes --with dev
Stack trace:
12 ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/cleo/application.py:327 in run
325│
326│ try:
→ 327│ exit_code = self._run(io)
328│ except BrokenPipeError:
329│ # If we are piped to another process, it may close early and send a
11 ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/console/application.py:190 in _run
188│ self._load_plugins(io)
189│
→ 190│ exit_code: int = super()._run(io)
191│ return exit_code
192│
10 ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/cleo/application.py:431 in _run
429│ io.input.interactive(interactive)
430│
→ 431│ exit_code = self._run_command(command, io)
432│ self._running_command = None
433│
9 ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/cleo/application.py:473 in _run_command
471│
472│ if error is not None:
→ 473│ raise error
474│
475│ return terminate_event.exit_code
8 ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/cleo/application.py:457 in _run_command
455│
456│ if command_event.command_should_run():
→ 457│ exit_code = command.run(io)
458│ else:
459│ exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED
7 ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/cleo/commands/base_command.py:119 in run
117│ io.input.validate()
118│
→ 119│ status_code = self.execute(io)
120│
121│ if status_code is None:
6 ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/cleo/commands/command.py:62 in execute
60│
61│ try:
→ 62│ return self.handle()
63│ except KeyboardInterrupt:
64│ return 1
5 ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry_plugin_export/command.py:107 in handle
105│ exporter.with_credentials(self.option("with-credentials"))
106│ exporter.with_urls(not self.option("without-urls"))
→ 107│ exporter.export(fmt, Path.cwd(), output or self.io)
108│
4 ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry_plugin_export/exporter.py:86 in export
84│ raise ValueError(f"Invalid export format: {fmt}")
85│
→ 86│ getattr(self, self.EXPORT_METHODS[fmt])(cwd, output)
87│
88│ def _export_generic_txt(
3 ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry_plugin_export/exporter.py:101 in _export_generic_txt
99│ )
100│
→ 101│ for dependency_package in get_project_dependency_packages(
102│ self._poetry.locker,
103│ project_requires=root.all_requires,
2 ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry_plugin_export/walker.py:103 in get_project_dependency_packages
101│ selected.append(dependency)
102│
→ 103│ for package, dependency in get_project_dependencies(
104│ project_requires=selected,
105│ locked_packages=repository.packages,
1 ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry_plugin_export/walker.py:129 in get_project_dependencies
127│ )
128│
→ 129│ nested_dependencies = walk_dependencies(
130│ dependencies=project_requires,
131│ packages_by_name=packages_by_name,
RuntimeError
Dependency walk failed at click (>=6.5)
at ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry_plugin_export/walker.py:155 in walk_dependencies
151│ requirement, packages_by_name, nested_dependencies
152│ )
153│
154│ if not locked_package:
→ 155│ raise RuntimeError(f"Dependency walk failed at {requirement}")
156│
157│ if requirement.extras:
158│ locked_package = locked_package.with_features(requirement.extras)
159│
You have not provided a pyproject.toml as requested in the issue template -- can you please provide that information so that people can help you debug?
I guess this is a duplicate of https://github.com/python-poetry/poetry-plugin-export/issues/173
@neersighted updated the issue description with pyproject.toml
@finswimmer i'm already using a python constraint so i don't believe this is a duplicate. Let me know.
It's hard to know what to make of this. So far as I can see poetry simply doesn't support source distributions with file suffix .tgz, which is all that is available at https://pypi.org/project/amqplib/1.0.2/#files, therefore the given pyproject.toml can't even lock.
So I don't know how this becomes an export problem...
I've updated the pyproject.toml file with packages that are the culprit - click and black.
Also, this issue might be related - https://github.com/psf/black/issues/2964, but i'm still not sure why export is not working?
Tweaked my pyproject.toml a bit more and was able to resolve the issue by specifying black versions for each python version.
Marking closed.
I don't know that this should be closed -- we shouldn't fail to export a valid lock file generated from a valid pyproject.
I'm having the same issue, not sure if it's the same cause. I think in our case it is related to #28. Our poetry.lock conatins duplicate entries for scipy (versions 1.8.1 and 1.10.0). This doesn't seem to be a problem, the poetry instally 1.10 and the project works fine. But export fails with Dependency walk failed at scipy (>=1.3,<1.9). I have no idea where the <1.9 constraint is coming from.
In our case a workaround is adding a direct scipy = ">=1.10" dependency in our pyproject.toml, but this seems artificial, since scipy is a subdependency, not a direct dependency for us.
Same as @mhamiltonj
In our case, the issue is https://github.com/statsmodels/statsmodels/issues/8543
Workaround: poetry add "statsmodels@^0.13.2,!=0.13.3,!=0.13.4,!=0.13.5"
Adding a direct dependency is not exactly a workaround; you should add constraints for anything you care to perturb/alter, not just anything you import directly. That being said, it is a workaround in this specific case as you are trying to perturb the lock file for us with this plugin, and not the dependency tree as a whole.
I encountered this Dependency walk failed at scipy (>=1.3,<1.9) error today.
I am able to reproduce this with this pyproject.toml and poetry.lock.
$ poetry debug info
Poetry
Version: 1.4.0
Python: 3.9.15
Virtualenv
Python: 3.9.15
Implementation: CPython
Path: /Users/colin/Library/Caches/pypoetry/virtualenvs/poetry-export-bug-gh176-qC8Y9brs-py3.9
Executable: /Users/colin/Library/Caches/pypoetry/virtualenvs/poetry-export-bug-gh176-qC8Y9brs-py3.9/bin/python
Valid: True
System
Platform: darwin
OS: posix
Python: 3.9.15
Path: /Users/colin/.pyenv/versions/3.9.15
Executable: /Users/colin/.pyenv/versions/3.9.15/bin/python3.9
N.b. it's got two scipy references in it: https://gist.github.com/colindean/e1b7326a567c44b5708cc822f9c7f561#file-poetry-lock-L1444-L1478
I think it might have something to do with the statsmodels package, which is coming into this through pyod. Specifically statsmodels spec for scipy:
scipy = [
{version = ">=1.3", markers = "python_version > \"3.9\" and python_version < \"3.12\" or platform_system != \"Windows\" and python_version < \"3.12\" or platform_machine != \"x86\" and python_version < \"3.12\""},
{version = ">=1.3,<1.9", markers = "python_version == \"3.9\" and platform_system == \"Windows\" and platform_machine == \"x86\""},
]
poetry export also appears to fail on poetry itself:
$ poetry debug info
Poetry
Version: 1.4.0
Python: 3.10.6
Virtualenv
Python: 3.10.6
Implementation: CPython
Path: /home/mlundy/.cache/pypoetry/virtualenvs/poetry-0f7nDIKJ-py3.10
Executable: /home/mlundy/.cache/pypoetry/virtualenvs/poetry-0f7nDIKJ-py3.10/bin/python
Valid: True
System
Platform: linux
OS: posix
Python: 3.10.6
Path: /usr
Executable: /usr/bin/python3.10
$ poetry self show plugins
• poetry-plugin-export (1.3.0) Poetry plugin to export the dependencies to various formats
1 application plugin
Dependencies
- poetry (>=1.3.0,<2.0.0)
- poetry-core (>=1.3.0,<2.0.0)
$ GIT_PAGER=cat git show --no-patch
commit 0e72a55c43a993ec0258facec23416c9212964ba (HEAD, tag: 1.4.0)
Author: Randy Döring <[email protected]>
Date: Sat Feb 18 04:19:46 2023
release: bump version to 1.4.0
$ poetry export -vvv
Loading configuration file /home/mlundy/.config/pypoetry/config.toml
Source (poetry-lockfile): 1 packages found for poetry-core 1.5.1
Source (poetry-lockfile): 1 packages found for poetry-plugin-export >=1.3.0,<2.0.0
Source (poetry-lockfile): 1 packages found for backports-cached-property >=1.0.2,<2.0.0
Source (poetry-lockfile): 1 packages found for build >=0.10.0,<0.11.0
Source (poetry-lockfile): 1 packages found for cachecontrol >=0.12.9,<0.13.0
Source (poetry-lockfile): 1 packages found for cleo >=2.0.0,<3.0.0
Source (poetry-lockfile): 1 packages found for crashtest >=0.4.1,<0.5.0
Source (poetry-lockfile): 1 packages found for dulwich >=0.21.2,<0.22.0
Source (poetry-lockfile): 1 packages found for filelock >=3.8.0,<4.0.0
Source (poetry-lockfile): 1 packages found for html5lib >=1.0,<2.0
Source (poetry-lockfile): 1 packages found for importlib-metadata >=4.4
Source (poetry-lockfile): 1 packages found for installer >=0.6.0,<0.7.0
Source (poetry-lockfile): 1 packages found for jsonschema >=4.10.0,<5.0.0
Source (poetry-lockfile): 1 packages found for keyring >=23.9.0,<24.0.0
Source (poetry-lockfile): 1 packages found for lockfile >=0.12.2,<0.13.0
Source (poetry-lockfile): 1 packages found for packaging >=20.4
Source (poetry-lockfile): 1 packages found for pexpect >=4.7.0,<5.0.0
Source (poetry-lockfile): 1 packages found for pkginfo >=1.9.4,<2.0.0
Source (poetry-lockfile): 1 packages found for platformdirs >=2.5.2,<3.0.0
Source (poetry-lockfile): 1 packages found for pyproject-hooks >=1.0.0,<2.0.0
Source (poetry-lockfile): 1 packages found for requests >=2.18,<3.0
Source (poetry-lockfile): 1 packages found for requests-toolbelt >=0.9.1,<0.11.0
Source (poetry-lockfile): 1 packages found for shellingham >=1.5,<2.0
Source (poetry-lockfile): 1 packages found for tomli >=2.0.1,<3.0.0
Source (poetry-lockfile): 1 packages found for tomlkit >=0.11.1,<0.11.2 || >0.11.2,<0.11.3 || >0.11.3,<1.0.0
Source (poetry-lockfile): 1 packages found for trove-classifiers >=2022.5.19
Source (poetry-lockfile): 2 packages found for virtualenv >=20.4.3,<20.4.5 || >20.4.5,<20.4.6 || >20.4.6,<21.0.0
Source (poetry-lockfile): 1 packages found for virtualenv <20.16.6
Source (poetry-lockfile): 1 packages found for xattr >=0.10.0,<0.11.0
Source (poetry-lockfile): 1 packages found for urllib3 >=1.26.0,<2.0.0
Stack trace:
12 ~/.local/pipx/venvs/poetry/lib/python3.10/site-packages/cleo/application.py:327 in run
325│
326│ try:
→ 327│ exit_code = self._run(io)
328│ except BrokenPipeError:
329│ # If we are piped to another process, it may close early and send a
11 ~/.local/pipx/venvs/poetry/lib/python3.10/site-packages/poetry/console/application.py:190 in _run
188│ self._load_plugins(io)
189│
→ 190│ exit_code: int = super()._run(io)
191│ return exit_code
192│
10 ~/.local/pipx/venvs/poetry/lib/python3.10/site-packages/cleo/application.py:431 in _run
429│ io.input.interactive(interactive)
430│
→ 431│ exit_code = self._run_command(command, io)
432│ self._running_command = None
433│
9 ~/.local/pipx/venvs/poetry/lib/python3.10/site-packages/cleo/application.py:473 in _run_command
471│
472│ if error is not None:
→ 473│ raise error
474│
475│ return terminate_event.exit_code
8 ~/.local/pipx/venvs/poetry/lib/python3.10/site-packages/cleo/application.py:457 in _run_command
455│
456│ if command_event.command_should_run():
→ 457│ exit_code = command.run(io)
458│ else:
459│ exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED
7 ~/.local/pipx/venvs/poetry/lib/python3.10/site-packages/cleo/commands/base_command.py:119 in run
117│ io.input.validate()
118│
→ 119│ status_code = self.execute(io)
120│
121│ if status_code is None:
6 ~/.local/pipx/venvs/poetry/lib/python3.10/site-packages/cleo/commands/command.py:62 in execute
60│
61│ try:
→ 62│ return self.handle()
63│ except KeyboardInterrupt:
64│ return 1
5 ~/.local/pipx/venvs/poetry/lib/python3.10/site-packages/poetry_plugin_export/command.py:107 in handle
105│ exporter.with_credentials(self.option("with-credentials"))
106│ exporter.with_urls(not self.option("without-urls"))
→ 107│ exporter.export(fmt, Path.cwd(), output or self.io)
108│
109│ return 0
4 ~/.local/pipx/venvs/poetry/lib/python3.10/site-packages/poetry_plugin_export/exporter.py:80 in export
78│ raise ValueError(f"Invalid export format: {fmt}")
79│
→ 80│ getattr(self, self.EXPORT_METHODS[fmt])(cwd, output)
81│
82│ def _export_generic_txt(
3 ~/.local/pipx/venvs/poetry/lib/python3.10/site-packages/poetry_plugin_export/exporter.py:95 in _export_generic_txt
93│ )
94│
→ 95│ for dependency_package in get_project_dependency_packages(
96│ self._poetry.locker,
97│ project_requires=root.all_requires,
2 ~/.local/pipx/venvs/poetry/lib/python3.10/site-packages/poetry_plugin_export/walker.py:97 in get_project_dependency_packages
95│ selected.append(dependency)
96│
→ 97│ for package, dependency in get_project_dependencies(
98│ project_requires=selected,
99│ locked_packages=repository.packages,
1 ~/.local/pipx/venvs/poetry/lib/python3.10/site-packages/poetry_plugin_export/walker.py:125 in get_project_dependencies
123│ )
124│
→ 125│ nested_dependencies = walk_dependencies(
126│ dependencies=project_requires,
127│ packages_by_name=packages_by_name,
RuntimeError
Dependency walk failed at virtualenv (<20.16.6)
at ~/.local/pipx/venvs/poetry/lib/python3.10/site-packages/poetry_plugin_export/walker.py:155 in walk_dependencies
151│ requirement, packages_by_name, nested_dependencies
152│ )
153│
154│ if not locked_package:
→ 155│ raise RuntimeError(f"Dependency walk failed at {requirement}")
156│
157│ if requirement.extras:
158│ locked_package = locked_package.with_features(requirement.extras)
159│
Just as @novas0x2a said - even poetry itself fails as it defines separate virtualenv dependency range for Windows and Python 3.9:
virtualenv = [
{version = ">=20.4.3,<20.4.5 || >20.4.5,<20.4.6 || >20.4.6,<21.0.0", markers = "sys_platform != \"win32\" or python_version != \"3.9\""},
{version = ">=20.4.3,<20.4.5 || >20.4.5,<20.4.6 || >20.4.6,<20.16.6", markers = "sys_platform == \"win32\" and python_version == \"3.9\""},
]
Closely related issues: https://github.com/python-poetry/poetry-plugin-export/issues/163, https://github.com/python-poetry/poetry-plugin-export/issues/168, https://github.com/python-poetry/poetry-plugin-export/issues/183, https://github.com/python-poetry/poetry-plugin-export/issues/186, https://github.com/python-poetry/poetry-plugin-export/issues/187.
Also, the culprit seems to be this block of code - https://github.com/python-poetry/poetry-plugin-export/blob/main/src/poetry_plugin_export/walker.py#L235. @finswimmer
I see this error on several newer packages:
- zipp 3.16.2+
- typing-extensions 4.8.0
when they are part of the dependency graph, poetry fails with "Dependency walk failed" error.
Anything came up since?
I'm using poetry==1.7.1
Edit: I've tried to add specific version of scipy and found that my python version (3.8) isn't supported for the latest scipy, which require minimum of 3.9
Hi, I need help, I have a similar issue, using poetry 1.8.2 and poetry-plugin-export (1.7.1).
poetry export --without-hashes --format=requirements.txt | sed 's#file:///.*/heyex-pipe/#file:///var/task/#g' > requirements.txt
Dependency walk failed at urllib3 (>=1.21.1,<3)
From my pyproject.toml:
[tool.poetry.dependencies]
python = ">=3.9,<3.11"
typed-argument-parser = "^1.7.2"
opencv-python = "4.6.0.66"
pillow = "^9.3.0"
tqdm = "^4.64.1"
scikit-image = "^0.19.3"
numpy = "^1.23.5"
tornado = "^6.2"
parse = "^1.19.0"
more-itertools = "^9.0.0"
matplotlib = "^3.7.0"
tensorflow = "2.13.1"
humanize = "^4.8.0"
torch = "2.0.1"
boto3 = "^1.34.0"
[tool.poetry.group.dev.dependencies]
pre-commit = "^2.20.0"
pytest-cov = "^4.0.0"
ipdb = "^0.13.11"
pdbpp = "^0.10.3"
boto3-stubs = { extras = ["lambda", "s3"], version = "^1.34.0" }
types-pillow = "^9.4.0.12"
types-tqdm = "^4.64.7.13"
pytest-xdist = "^3.2.0"
keras = "^2.11.0"
onnxruntime-gpu = "^1.13.1"
pip = "^23.1.2"
wheel = "^0.40.0"
ruff = "^0.1.3"
mypy = "^1.6.1"
types-requests = "2.31.0.6"
mypy-boto3-lambda = "^1.34.0"
mypy-boto3-s3 = "^1.34.0"
Never mind, the issue is the way botocore deals with dependencies. Upgraded to Python 3.10 and it seems to work now.
