poetry
poetry copied to clipboard
Published
20 hours ago •
python-poetry
python-poetry
[Windows] Installing a package from git is leading to "No such file or directory" error
Description
When I install a package from git on windows by doing poetry add git+https://github.com/BerriAI/litellm.git, I get an error:
[Errno 2] No such file or directory: b'<path to the python env>\\src\\litellm\\.git\\refs\\remotes\\origin\\dependabot\\npm_and_yarn\\docs\\my-website\\trim-and-docusaurus\\core-and-docusaurus\\plugin-google-gtag-and-docusaurus\\plugin-ideal-image-and-docusaurus\\preset-classic--removed.lock'
Where <path to the python env> is just a placeholder for a longer base path to my python environment.
If I do the same on Mac, it works without issues.
What I think is, that the path is just too long for windows. Because Windows only supports path up to 260 characters. If you want to use paths longer than that you have to use the prefix \\?\, which allows path to up to 32.000 characters. See: https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry
If I run the command with -vvv, I get the following output:
Using virtualenv: <path to the python env>
Checking if keyring is available
[keyring:keyring.backend] Loading KWallet
[keyring:keyring.backend] Loading SecretService
[keyring:keyring.backend] Loading Windows
[win32ctypes:win32ctypes.core.cffi] Loaded cffi backend
[keyring:keyring.backend] Loading chainer
[keyring:keyring.backend] Loading libsecret
[keyring:keyring.backend] Loading macOS
Using keyring backend 'Windows WinVaultKeyring'
[urllib3:urllib3.connectionpool] Starting new HTTPS connection (1): github.com:443
[urllib3:urllib3.connectionpool] https://github.com:443 "GET /BerriAI/litellm.git/info/refs?service=git-upload-pack HTTP/1.1" 200 None
Cloning https://github.com/BerriAI/litellm.git at 'HEAD' to <path to the python env>\src\litellm
Stack trace:
20 <path to the python env>\Lib\site-packages\cleo\application.py:327 in run
325│
326│ try:
→ 327│ exit_code = self._run(io)
328│ except BrokenPipeError:
329│ # If we are piped to another process, it may close early and send a
19 <path to the python env>\Lib\site-packages\poetry\console\application.py:190 in _run
188│ self._load_plugins(io)
189│
→ 190│ exit_code: int = super()._run(io)
191│ return exit_code
192│
18 <path to the python env>\Lib\site-packages\cleo\application.py:431 in _run
429│ io.input.interactive(interactive)
430│
→ 431│ exit_code = self._run_command(command, io)
432│ self._running_command = None
433│
17 <path to the python env>\Lib\site-packages\cleo\application.py:473 in _run_command
471│
472│ if error is not None:
→ 473│ raise error
474│
475│ return terminate_event.exit_code
16 <path to the python env>\Lib\site-packages\cleo\application.py:457 in _run_command
455│
456│ if command_event.command_should_run():
→ 457│ exit_code = command.run(io)
458│ else:
459│ exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED
15 <path to the python env>\Lib\site-packages\cleo\commands\base_command.py:117 in run
115│ io.input.validate()
116│
→ 117│ return self.execute(io) or 0
118│
119│ def merge_application_definition(self, merge_args: bool = True) -> None:
14 <path to the python env>\Lib\site-packages\cleo\commands\command.py:61 in execute
59│
60│ try:
→ 61│ return self.handle()
62│ except KeyboardInterrupt:
63│ return 1
13 <path to the python env>\Lib\site-packages\poetry\console\commands\add.py:164 in handle
162│ return 0
163│
→ 164│ requirements = self._determine_requirements(
165│ packages,
166│ allow_prereleases=self.option("allow-prereleases"),
12 <path to the python env>\Lib\site-packages\poetry\console\commands\init.py:375 in _determine_requirements
373│
374│ result = []
→ 375│ for requirement in self._parse_requirements(requires):
376│ if "git" in requirement or "url" in requirement or "path" in requirement:
377│ result.append(requirement)
11 <path to the python env>\Lib\site-packages\poetry\console\commands\init.py:441 in _parse_requirements
439│ cwd=cwd,
440│ )
→ 441│ return [parser.parse(requirement) for requirement in requirements]
442│
443│ def _format_requirements(self, requirements: list[dict[str, str]]) -> Requirements:
10 <path to the python env>\Lib\site-packages\poetry\utils\dependency_specification.py:89 in parse
87│
88│ specification = (
→ 89│ self._parse_url(requirement)
90│ or self._parse_path(requirement)
91│ or self._parse_simple(requirement)
9 <path to the python env>\Lib\site-packages\poetry\utils\dependency_specification.py:149 in _parse_url
147│
148│ if url_parsed.scheme in GIT_URL_SCHEMES:
→ 149│ return self._parse_git_url(requirement)
150│
151│ if url_parsed.scheme in ["http", "https"]:
8 <path to the python env>\Lib\site-packages\poetry\utils\dependency_specification.py:133 in _parse_git_url
131│
132│ source_root = self._env.path.joinpath("src") if self._env else None
→ 133│ package = self._direct_origin.get_package_from_vcs(
134│ "git",
135│ url=url.url,
7 <path to the python env>\llmaas\Lib\site-packages\poetry\packages\direct_origin.py:106 in get_package_from_vcs
104│ raise ValueError(f"Unsupported VCS dependency {vcs}")
105│
→ 106│ return _get_package_from_git(
107│ url=url,
108│ branch=branch,
6 <path to the python env>\Lib\site-packages\poetry\packages\direct_origin.py:32 in _get_package_from_git
30│ source_root: Path | None = None,
31│ ) -> Package:
→ 32│ source = Git.clone(
33│ url=url,
34│ source_root=source_root,
5 <path to the python env>\Lib\site-packages\poetry\vcs\git\backend.py:455 in clone
453│ try:
454│ if not cls.is_using_legacy_client():
→ 455│ local = cls._clone(url=url, refspec=refspec, target=target)
456│ cls._clone_submodules(repo=local)
457│ return local
4 <path to the python env>\Lib\site-packages\poetry\vcs\git\backend.py:288 in _clone
286│ (b"refs/tags", b"refs/tags"),
287│ }:
→ 288│ local.refs.import_refs(
289│ base=base,
290│ other={
3 <path to the python env>\Lib\site-packages\dulwich\refs.py:189 in import_refs
187│ to_delete.add(name)
188│ else:
→ 189│ self.set_if_equals(
190│ b"/".join((base, name)), None, value, message=message
191│ )
2 <path to the python env>\Lib\site-packages\dulwich\refs.py:897 in set_if_equals
895│
896│ ensure_dir_exists(os.path.dirname(filename))
→ 897│ with GitFile(filename, "wb") as f:
898│ if old_ref is not None:
899│ try:
1 <path to the python env>\Lib\site-packages\dulwich\file.py:93 in GitFile
91│ raise OSError("text mode not supported for Git files")
92│ if "w" in mode:
→ 93│ return _GitFile(filename, mode, bufsize, mask)
94│ else:
95│ return open(filename, mode, bufsize)
FileNotFoundError
[Errno 2] No such file or directory: b'<path to the python env>\\src\\litellm\\.git\\refs\\remotes\\origin\\dependabot\\npm_and_yarn\\docs\\my-website\\trim-and-docusaurus\\core-and-docusaurus\\plugin-google-gtag-and-docusaurus\\plugin-ideal-image-and-docusaurus\\preset-classic--removed.lock'
at <path to the python env>\Lib\site-packages\dulwich\file.py:149 in __init__
145│ self._lockfilename = self._filename + b".lock"
146│ else:
147│ self._lockfilename = self._filename + ".lock"
148│ try:
→ 149│ fd = os.open(
150│ self._lockfilename,
151│ os.O_RDWR | os.O_CREAT | os.O_EXCL | getattr(os, "O_BINARY", 0),
152│ mask,
153│ )
I think poetry must add the prefix for long windows paths to the source_root (or base path) where it wants the git clone operation to take part.
In my code, which needs to run on different systems, I often use a function like that:
def adapt_for_long_windows_path_if_necessary(path: str) -> str:
if os.name == "nt": # pragma: no cover - system dependent
path = '\\\\?\\' + str(Path(path).absolute().resolve()) # pragma: no cover - system dependent
return path
Here it is necessary to make the path absolute, since the prefix only works with absolute paths.
Workarounds
I can install the package by pip directly and run operations like poetry lock on a linux client. But this is not a workaround for everyone.
Poetry Installation Method
pip
Operating System
Windows 11
Poetry Version
Poetry (version 1.8.3)
Poetry Configuration
cache-dir = "<a path in my user directory>"
experimental.system-git-client = false
installer.max-workers = null
installer.modern-installation = true
installer.no-binary = null
installer.parallel = true
keyring.enabled = true
repositories.<private-repro>.url = "<url to private repro>"
solver.lazy-wheel = true
virtualenvs.create = true
virtualenvs.in-project = null
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}\\virtualenvs" # <a path to my user directory>
virtualenvs.prefer-active-python = false
virtualenvs.prompt = "{project_name}-py{python_version}"
warnings.export = true
Python Sysconfig
Platform: "win-amd64"
Python version: "3.12"
Current installation scheme: "nt"
Paths:
data = "<path to the python env>"
include = "<path to the python env>\Include"
platinclude = "<path to the python env>\Include"
platlib = "<path to the python env>\Lib\site-packages"
platstdlib = "<path to the python env>\Lib"
purelib = "<path to the python env>\Lib\site-packages"
scripts = "<path to the python env>\Scripts"
stdlib = "<path to the python env>\Lib"
Variables:
BINDIR = "<path to the python env>"
BINLIBDEST = "<path to the python env>\Lib"
EXE = ".exe"
EXT_SUFFIX = ".cp312-win_amd64.pyd"
INCLUDEPY = "<path to the python env>\Include"
LIBDEST = "<path to the python env>\Lib"
TZPATH = "<path to the python env>\share\zoneinfo"
VERSION = "312"
VPATH = "..\.."
abiflags = ""
base = "<path to the python env>"
exec_prefix = "<path to the python env>"
installed_base = "<path to the python env>"
installed_platbase = "<path to the python env>"
platbase = "<path to the python env>"
platlibdir = "DLLs"
prefix = "<path to the python env>"
projectbase = "<path to the python env>"
py_version = "3.12.3"
py_version_nodot = "312"
py_version_nodot_plat = "312"
py_version_short = "3.12"
srcdir = "<path to the python env>"
userbase = "<path to my user directory>\AppData\Roaming\Python"
Example pyproject.toml
[tool.poetry]
name = "test"
version = "0.1.0"
description = ""
authors = ["me"]
readme = "README.md"
[tool.poetry.dependencies]
python = "^3.12"
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
Poetry Runtime Logs
see above
