python-poetry
Git repository and package name mismatch
- Poetry version: 1.2.2
- Python version: 3.8.2
- OS version and name: macOS Monterey
- pyproject.toml: https://gist.github.com/antoineeudes/dfccd21bb13dd5d55e8bf434b7d59cdd
- [x] I am on the latest stable Poetry version, installed using a recommended method.
- [x] I have searched the issues of this repo and believe that this is not a duplicate.
- [x] I have consulted the FAQ and blog for any relevant entries or release notes.
- [x] If an exception occurs when executing a command, I executed it again in debug mode (
-vvvoption) and have included the output below.
Issue
Hello Poetry team!
I am trying to install a dependency from a private Gitlab repository. Let's call the repository foo and the library bar.
In my pyproject.toml, I added the following line:
bar = {git = "ssh://[email protected]/xxx/foo.git", develop = true, tag = "v0.13.1"}
After executing poetry install, I see that the library is successfully installed:
~ poetry run python
Python 3.8.2 (v3.8.2:7b3ab5921f, Feb 24 2020, 17:52:18)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import bar
>>> bar.__version__
'v0.13.1'
But, when I try to install another dependency, I get an error:
~ poetry add pandas
No git repository was found at /Users/me/Library/Caches/pypoetry/virtualenvs/xxxxxxxxx-py3.8/src/bar
It appears the git repository is not found in the virtual environment under the path src/bar. However, I can see the repo is there under the path src/foo. I guess there could be a confusion between the name of the library and the name of the git repository.
~ ls /Users/me/Library/Caches/pypoetry/virtualenvs/xxxxxxxxx-py3.8/src/
foo
Here is the complete log:
~ poetry -vvv add pandas
Loading configuration file /Users/me/Library/Preferences/pypoetry/config.toml
Using virtualenv: /Users/me/Library/Caches/pypoetry/virtualenvs/xxxxxxxxx-py3.8
Stack trace:
18 ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/cleo/application.py:329 in run
327│
328│ try:
→ 329│ exit_code = self._run(io)
330│ except Exception as e:
331│ if not self._catch_exceptions:
17 ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/poetry/console/application.py:185 in _run
183│ self._load_plugins(io)
184│
→ 185│ exit_code: int = super()._run(io)
186│ return exit_code
187│
16 ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/cleo/application.py:423 in _run
421│ io.input.set_stream(stream)
422│
→ 423│ exit_code = self._run_command(command, io)
424│ self._running_command = None
425│
15 ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/cleo/application.py:465 in _run_command
463│
464│ if error is not None:
→ 465│ raise error
466│
467│ return event.exit_code
14 ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/cleo/application.py:446 in _run_command
444│
445│ try:
→ 446│ self._event_dispatcher.dispatch(event, COMMAND)
447│
448│ if event.command_should_run():
13 ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/cleo/events/event_dispatcher.py:23 in dispatch
21│
22│ if listeners:
→ 23│ self._do_dispatch(listeners, event_name, event)
24│
25│ return event
12 ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/cleo/events/event_dispatcher.py:84 in _do_dispatch
82│ break
83│
→ 84│ listener(event, event_name, self)
85│
86│ def _sort_listeners(self, event_name: str) -> None:
11 ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/poetry/console/application.py:316 in configure_installer_for_event
314│ return
315│
→ 316│ cls.configure_installer_for_command(command, event.io)
317│
318│ @staticmethod
10 ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/poetry/console/application.py:323 in configure_installer_for_command
321│
322│ poetry = command.poetry
→ 323│ installer = Installer(
324│ io,
325│ command.env,
9 ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/poetry/installation/installer.py:79 in __init__
77│ self._installer = self._get_installer()
78│ if installed is None:
→ 79│ installed = self._get_installed()
80│
81│ self._installed_repository = installed
8 ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/poetry/installation/installer.py:573 in _get_installed
571│
572│ def _get_installed(self) -> InstalledRepository:
→ 573│ return InstalledRepository.load(self._env)
574│
7 ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/poetry/repositories/installed_repository.py:291 in load
289│ continue
290│
→ 291│ package = cls.create_package_from_distribution(distribution, env)
292│
293│ if with_dependencies:
6 ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/poetry/repositories/installed_repository.py:158 in create_package_from_distribution
156│ source_url,
157│ source_reference,
→ 158│ ) = cls.get_package_vcs_properties_from_path(
159│ env.path / "src" / canonicalize_name(distribution.metadata["name"])
160│ )
5 ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/poetry/repositories/installed_repository.py:89 in get_package_vcs_properties_from_path
87│ from poetry.vcs.git import Git
88│
→ 89│ info = Git.info(repo=src)
90│ return "git", info.origin, info.revision
91│
4 ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/poetry/vcs/git/backend.py:174 in info
172│ @classmethod
173│ def info(cls, repo: Repo | Path | str) -> GitRepoLocalInfo:
→ 174│ return GitRepoLocalInfo(repo=repo)
175│
176│ @staticmethod
3 <string>:3 in __init__
1│
2 ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/poetry/vcs/git/backend.py:144 in __post_init__
142│
143│ def __post_init__(self, repo: Repo | Path | str) -> None:
→ 144│ repo = Git.as_repo(repo=repo) if not isinstance(repo, Repo) else repo
145│ self.origin = Git.get_remote_url(repo=repo, remote="origin")
146│ self.revision = Git.get_revision(repo=repo)
1 ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/poetry/vcs/git/backend.py:152 in as_repo
150│ @staticmethod
151│ def as_repo(repo: Path | str) -> Repo:
→ 152│ return Repo(str(repo))
153│
154│ @staticmethod
NotGitRepository
No git repository was found at /Users/me/Library/Caches/pypoetry/virtualenvs/xxxxxxxxx-py3.8/src/bar
at ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/dulwich/repo.py:1126 in __init__
1122│ elif (os.path.isdir(os.path.join(root, OBJECTDIR))
1123│ and os.path.isdir(os.path.join(root, REFSDIR))):
1124│ bare = True
1125│ else:
→ 1126│ raise NotGitRepository(
1127│ "No git repository was found at %(path)s" % dict(path=root)
1128│ )
1129│
1130│ self.bare = bare
Thank you !
I updated the title to describe what is actually going on; overall I think our VCS support breaks when subdirectories or mismatched names are involved, and this may be a duplicate of an existing issue. Not sure who to ping on VCS issues, but this entire subject could use some eyes, so I'll try to bother the other maintainers about it.
This got broken with some version between 1.1.14 and 1.2.2
Maybe by introducing canonicalize_name with 1.2.0: https://github.com/python-poetry/poetry/issues/4770#issuecomment-1179560514
The same happens with the dash/underscore transformation/mapping:
poetry 1.1.14
# pyproject.toml
abas-erp-api-client = {git = "ssh://[email protected]/abas_erp_api_client.git", develop = true, rev = "main"}
$ poetry install
$ ls venv/src
abas-erp-api-client
$ poetry update
No dependencies to install or update
poetry 1.2.2
# pyproject.toml
abas-erp-api-client = {git = "ssh://[email protected]/abas_erp_api_client.git", develop = true, rev = "main"}
$ poetry install
$ ls venv/src
abas_erp_api_client
$ poetry update
No git repository was found at /Users/maltegerth/test-poetry-1.2.2/venv/src/abas-erp-api-client
This got broken with some version between 1.1.14 and 1.2.2
I don't think this assertion is correct per-se; mismatched names have always been an issue. We've added more features that can trigger this, and made our name handling code more robust. But I think this has always been fundamentally naive/broken, and recent changes have just increased the surface area.
Here's a possible workaround for anyone else hitting this.
I ran into this while trying to upgrade from Poetry 1.1 and I found poetry install only fails when the dependency has develop = true. So I removed develop = true in pyproject.toml, manually updated develop = false in poetry.lock, and poetry env remove'd the broken env. Now poetry install works every time for Poetry 1.2+.
This does mean we can't use develop = true. For anyone that needs to develop locally, this is definitely not ideal. But it should be fine if you're just pinning to git commits to work around release issues (like me), because you're not actually working on the dependencies locally. (I don't actually know why we committed develop = true to our repo in the first place. Persisting a local dev flag seems like a bug. Unless I'm missing something about git deps?)
All four of these issues appear to be the same issue (AFAICT):
- https://github.com/python-poetry/poetry/issues/6958
- https://github.com/python-poetry/poetry/issues/7113 (this issue)
- https://github.com/python-poetry/poetry/issues/7181
- https://github.com/python-poetry/poetry/issues/7949
Of the four, this issue (#7113) seems to get to the essence of the fundamental problem, nicely summarized by @antoineeudes :
I guess there could be a confusion between the name of the library and the name of the git repository.
I would guess that this could be solved with the "Potential fix" described here: https://github.com/python-poetry/poetry/issues/7181#issuecomment-1427120155
(... but I haven't worked on Poetry, so perhaps that's naive)
I'm guessing that a significant number of users are running into this problem since there are so many different issues about it, and it likely prevents anyone from using Poetry with any sort of private git monorepos (or even just repos with more than one package).
I can't seem to reproduce it on Poetry 1.5.1. Here's my pyproject.toml
[tool.poetry]
name = "poetry-git"
version = "0.1.0"
description = ""
authors = ["Riccardo Albertazzi <[email protected]>"]
readme = "README.md"
packages = [{include = "poetry_git"}]
[tool.poetry.dependencies]
python = "^3.9"
# A plain simple fork of requests
requests = {git = "https://github.com/ralbertazzi/rfork.git", develop = true}
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
every command succeeds.
❯ poetry install
Creating virtualenv poetry-git in /<path-to-project>/.venv
Installing dependencies from lock file
Package operations: 5 installs, 0 updates, 0 removals
• Installing certifi (2023.5.7)
• Installing charset-normalizer (3.1.0)
• Installing idna (3.4)
• Installing urllib3 (2.0.2)
• Installing requests (2.31.0 6e5b15d)
Installing the current project: poetry-git (0.1.0)
❯ poetry update
Updating dependencies
Resolving dependencies... (1.6s)
Package operations: 0 installs, 1 update, 0 removals
• Updating requests (2.31.0 /<path-to-project>/.venv/src/rfork -> 2.31.0 6e5b15d)
❯ poetry add redis
Using version ^4.5.5 for redis
Updating dependencies
Resolving dependencies... (0.1s)
Package operations: 2 installs, 1 update, 0 removals
• Installing async-timeout (4.0.2)
• Installing redis (4.5.5)
• Updating requests (2.31.0 /<path-to-project>/.venv/src/rfork -> 2.31.0 6e5b15d)
Here's the poetry.lock entry for reference:
[[package]]
name = "requests"
version = "2.31.0"
description = "Python HTTP for Humans."
optional = false
python-versions = ">=3.7"
files = []
develop = true
[package.dependencies]
certifi = ">=2017.4.17"
charset_normalizer = ">=2,<4"
idna = ">=2.5,<4"
urllib3 = ">=1.21.1,<3"
[package.extras]
socks = ["PySocks (>=1.5.6,!=1.5.7)"]
use-chardet-on-py3 = ["chardet (>=3.0.2,<6)"]
[package.source]
type = "git"
url = "https://github.com/ralbertazzi/rfork.git"
reference = "HEAD"
resolved_reference = "6e5b15d542a4e85945fd72066bb6cecbc3a82191"
I'd be great if you could provide a way to reproduce the issue on the latest Poetry
For doubt's sake I also tried the ssh specification and that also works:
requests = {git = "ssh://[email protected]/ralbertazzi/rfork.git", develop = true}
@ralbertazzi can you try poetry install with this repo https://github.com/JonathanRayner/some_other_repo ? Here I have 5 packages that are each slower clones, hopefully more consistently triggering the issue.
