poetry throws an error about SHA string length when trying to install a dep from a private repo using a token

[sg-s]

• Poetry version: Poetry (version 1.4.1)

• Python version: Python: 3.9.15

• OS version and name: macOS 13.2.1

• pyproject.toml: https://gist.github.com/sg-s/07ae0c878981d15b5c22d2adc7d74f3d

• [x] I am on the latest stable Poetry version, installed using a recommended method.

• [x] I have searched the issues of this repo and believe that this is not a duplicate.

• [x] I have consulted the FAQ and blog for any relevant entries or release notes.

• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

what i'm trying to do

• install a dependency that is a private repo on github, using a github token
• i did the following:

poetry -vvv add git+https://{token}@github.com/my/private-repo/

what i expected to happen

it to install, or at least give me some error about access.

what i got instead

an error that the length of SHA string isn't 20

base error

  AssertionError

  

  at ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/dulwich/pack.py:699 in _object_offset
       695│ 
       696│         Args:
       697│           sha: A *binary* SHA string. (20 characters long)_
       698│         """
    →  699│         assert len(sha) == 20
       700│         idx = ord(sha[:1])
       701│         if idx == 0:
       702│             start = 0
       703│         else:

detailed error

Using virtualenv: /Users/srinivas/Library/Caches/pypoetry/virtualenvs/ideas-python-utils-ibgcsbwi-py3.9
[keyring.backend] Loading KWallet
[keyring.backend] Loading SecretService
[keyring.backend] Loading Windows
[keyring.backend] Loading chainer
[keyring.backend] Loading libsecret
[keyring.backend] Loading macOS

  Stack trace:

  26  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/cleo/application.py:327 in run
       325│ 
       326│             try:
     → 327│                 exit_code = self._run(io)
       328│             except BrokenPipeError:
       329│                 # If we are piped to another process, it may close early and send a

  25  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/console/application.py:190 in _run
       188│         self._load_plugins(io)
       189│ 
     → 190│         exit_code: int = super()._run(io)
       191│         return exit_code
       192│ 

  24  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/cleo/application.py:431 in _run
       429│             io.input.interactive(interactive)
       430│ 
     → 431│         exit_code = self._run_command(command, io)
       432│         self._running_command = None
       433│ 

  23  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/cleo/application.py:473 in _run_command
       471│ 
       472│         if error is not None:
     → 473│             raise error
       474│ 
       475│         return terminate_event.exit_code

  22  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/cleo/application.py:457 in _run_command
       455│ 
       456│             if command_event.command_should_run():
     → 457│                 exit_code = command.run(io)
       458│             else:
       459│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

  21  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/cleo/commands/base_command.py:119 in run
       117│         io.input.validate()
       118│ 
     → 119│         status_code = self.execute(io)
       120│ 
       121│         if status_code is None:

  20  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/cleo/commands/command.py:62 in execute
        60│ 
        61│         try:
     →  62│             return self.handle()
        63│         except KeyboardInterrupt:
        64│             return 1

  19  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/console/commands/add.py:160 in handle
       158│             return 0
       159│ 
     → 160│         requirements = self._determine_requirements(
       161│             packages,
       162│             allow_prereleases=self.option("allow-prereleases"),

  18  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/console/commands/init.py:384 in _determine_requirements
       382│ 
       383│         result = []
     → 384│         for requirement in self._parse_requirements(requires):
       385│             if "git" in requirement or "url" in requirement or "path" in requirement:
       386│                 result.append(requirement)

  17  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/console/commands/init.py:443 in _parse_requirements
       441│             cwd = Path.cwd()
       442│ 
     → 443│         return [
       444│             parse_dependency_specification(
       445│                 requirement=requirement,

  16  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/console/commands/init.py:444 in <listcomp>
       442│ 
       443│         return [
     → 444│             parse_dependency_specification(
       445│                 requirement=requirement,
       446│                 env=self.env if isinstance(self, EnvCommand) else None,

  15  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/utils/dependency_specification.py:216 in parse_dependency_specification
       214│ 
       215│     specification = (
     → 216│         _parse_dependency_specification_url(requirement, env=env)
       217│         or _parse_dependency_specification_path(requirement, cwd=cwd)
       218│         or _parse_dependency_specification_simple(requirement)

  14  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/utils/dependency_specification.py:68 in _parse_dependency_specification_url
        66│ 
        67│     if url_parsed.scheme in ["git+https", "git+ssh"]:
     →  68│         return _parse_dependency_specification_git_url(requirement, env)
        69│ 
        70│     if url_parsed.scheme in ["http", "https"]:

  13  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/utils/dependency_specification.py:49 in _parse_dependency_specification_git_url
        47│ 
        48│     source_root = env.path.joinpath("src") if env else None
     →  49│     package = Provider.get_package_from_vcs(
        50│         "git",
        51│         url=url.url,

  12  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/puzzle/provider.py:384 in get_package_from_vcs
        382│             raise ValueError(f"Unsupported VCS dependency {vcs}")
        383│ 
     →  384│         return _get_package_from_git(
        385│             url=url,
        386│             branch=branch,

  11  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/puzzle/provider.py:112 in _get_package_from_git
        110│     source_root: Path | None = None,
        111│ ) -> Package:
     →  112│     source = Git.clone(
        113│         url=url,
        114│         source_root=source_root,

  10  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/vcs/git/backend.py:429 in clone
       427│         try:
       428│             if not cls.is_using_legacy_client():
     → 429│                 local = cls._clone(url=url, refspec=refspec, target=target)
       430│                 cls._clone_submodules(repo=local)
       431│                 return local

   9  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/vcs/git/backend.py:256 in _clone
       254│             local = Repo(str(target))
       255│ 
     → 256│         remote_refs = cls._fetch_remote_refs(url=url, local=local)
       257│ 
       258│         logger.debug(

   8  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/vcs/git/backend.py:201 in _fetch_remote_refs
       199│ 
       200│         with local:
     → 201│             result: FetchPackResult = client.fetch(
       202│                 path,
       203│                 local,

   7  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/dulwich/client.py:785 in fetch
        783│                 path,
        784│                 determine_wants,
     →  785│                 target.get_graph_walker(),
        786│                 f.write,
        787│                 progress=progress,

   6  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/dulwich/repo.py:554 in get_graph_walker
        552│         """
        553│         if heads is None:
     →  554│             heads = [
        555│                 sha
        556│                 for sha in self.refs.as_dict(b"refs/heads").values()

   5  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/dulwich/repo.py:557 in <listcomp>
        555│                 sha
        556│                 for sha in self.refs.as_dict(b"refs/heads").values()
     →  557│                 if sha in self.object_store
        558│             ]
        559│         parents_provider = ParentsProvider(self.object_store)

   4  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/dulwich/object_store.py:385 in __contains__
        383│         This method makes no distinction between loose and packed objects.
        384│         """
     →  385│         if self.contains_packed(sha) or self.contains_loose(sha):
        386│             return True
        387│         for alternate in self.alternates:

   3  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/dulwich/object_store.py:374 in contains_packed
        372│         for pack in self.packs:
        373│             try:
     →  374│                 if sha in pack:
        375│                     return True
        376│             except PackFileDisappeared:

   2  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/dulwich/pack.py:2370 in __contains__
       2368│         """Check whether this pack contains a particular SHA1."""
       2369│         try:
     → 2370│             self.index.object_offset(sha1)
       2371│             return True
       2372│         except KeyError:

   1  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/dulwich/pack.py:686 in object_offset
        684│             sha = hex_to_sha(sha)
        685│         try:
     →  686│             return self._object_offset(sha)
        687│         except ValueError as exc:
        688│             closed = getattr(self._contents, "closed", None)

  AssertionError

  

  at ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/dulwich/pack.py:699 in _object_offset
       695│ 
       696│         Args:
       697│           sha: A *binary* SHA string. (20 characters long)_
       698│         """
    →  699│         assert len(sha) == 20
       700│         idx = ord(sha[:1])
       701│         if idx == 0:
       702│             start = 0
       703│         else:

[dimbleby]

presumably duplicate #6455, #7596, though without a way to repro no-one can do anything with this anyway

please close

[sg-s]

it's pretty easy to reproduce:

poetry add git+https://<any private repo>

[sg-s]

and @dimbleby while its true that i get the same error, using longer hashes doens't work

[dimbleby]

I did a poetry add with a private repo that I have on github and this did not reproduce. Without a way to reproduce, no-one but you can do anything with this.

But it's still bound to be a duplicate of those others and can be closed as such.

[sg-s]

@dimbleby can i ask what command you used?

i'd love to reproduce your success

[dimbleby]

exactly as you suggested:

poetry add git+https://ghp_<redacted>@github.com:dimbleby/<redacted>

[sg-s]

and did that private git repo dependency have a pyproject.toml file (mine does not)

[dimbleby]

this is a terrible way to debug and I am dropping out of the conversation.

I recommend that you close this as a duplicate of one of the others.

If you want to share your problem, then set up a repository that does not contain anything secret so that anyone can reproduce whatever it is you are seeing.

[timos-flwls]

I've also run into this problem, I don't have a reproduction, but I can leave a breadcrumb ... it seems to be triggered when a repo that your other private repo depends on is already cloned into Poetry's .venv/src folder. Deleting the specific repo that your private repo depends on from that folder and adding it again via the normal command seems to run.

[DrewDevereux]

Encountered this error today. Poetry 1.3.2, installing from an open source git repo, not using a token. But also on macOS. We were installing into a Docker devcontainer, and we got around the problem by deleting the lock file then rebuilding the container i.e. resolving afresh against an empty cache / env.

[aqeelat]

@dimbleby continuing our discussion in #8112,

What more info do you need to reproduce? Also, if I wanted to live-debug poetry, would I be able to use pdb?

[dimbleby]

What more info do you need to reproduce?

... a way to reproduce it!

If you can start from a fresh project and provide steps to show the error, please do so. Even better, wrap that up in a dockerfile to make sure that the same thing will happen for someone else.

poetry is just a regular python program, debug it however you like

[emanuele]

I've also run into this problem, I don't have a reproduction, but I can leave a breadcrumb ... it seems to be triggered when a repo that your other private repo depends on is already cloned into Poetry's .venv/src folder. Deleting the specific repo that your private repo depends on from that folder and adding it again via the normal command seems to run.

I had a similar issue: poetry lock always failed with the exception mentioned in this issue whenever a given dependency from a public git repo (pytensor) was in pyproject.toml. So, in my case, it happened from a public repo (not PRIVATE!). The solution was the one quoted, thanks @timos-flwls : I just deleted .venv/src/pytensor and the problem disappeared.

Maybe, in my case, an initial poetry install was interrupted and left a corrupted/broken clone in .venv/src/pytensor and so all subsequent poetry lock failed. Clearly, this is hard to reproduce, unless you damage a previously cloned repo on purpose.

Maybe a dulwich issue more than a poetry issue?

All in all: remove the guilty repo in .venv/src/ if you stumble into this issue.

[sacundim]

I just experienced this error while doing a poetry lock with a public git repo and, similar to what @emanuele says, in my case it went away after I nuked the venv and recreated it.

  AssertionError

  at ~/.local/pipx/venvs/poetry/lib/python3.11/site-packages/dulwich/pack.py:698 in _object_offset
       694│ 
       695│         Args:
       696│           sha: A *binary* SHA string. (20 characters long)_
       697│         """
    →  698│         assert len(sha) == 20
       699│         idx = ord(sha[:1])
       700│         if idx == 0:
       701│             start = 0
       702│         else:

[bobokvsky]

In my way, it reproduced like that:

1. In repo: make commit, push it (with some commit hash)
2. In poetry project: add repo {git = "path-to-repo", rev = "hash"}, run poetry lock
3. In repo make git reset HEAD~1, then make new commit on base of it, then force push the commit git push -f (it has some commit with new hash2)
4. In poetry project: change repo {git = "path-to-repo", rev = "hash2"}, run poetry lock, it has the error assert len(sha) == 20

I assume that older hash is deleted when we made a force push commit, so poetry couldn't detect an older hash in github to compute SHA correctly. As it's written above, removing .venv/ solves the problem

[gogobera]

Bug and resolution still exist with Poetry 1.6.1. Removing repo in ~/.cache/pypoetry/virtualenvs/<redacted>-py3.11/src allowed the poetry lock command to succeed. Error was consistent regardless of using the git repo's full hash, the 11 char slug, or the first 20 chars. Maddening. Glad this thread exists.

[techwizrd]

I can confirm this bug still occurs in Poetry 1.7.1 as well. Maddening indeed.

[dbalabka]

Even more strange behaviour. I tried to install it with short SHA, and it failed on the first try and succeeded only on the second:

❯ poetry add "git+https://github.com/dask/dask-kubernetes.git#dfeca2b"

Failed to clone https://github.com/dask/dask-kubernetes.git at 'dfeca2b', verify ref exists on remote.

❯ poetry add "git+https://github.com/dask/dask-kubernetes.git#dfeca2b38fe504692091c153d297010004746aa7"

  AssertionError



  at ~/.local/pipx/venvs/poetry/lib/python3.10/site-packages/dulwich/pack.py:742 in _object_offset
       738│
       739│         Args:
       740│           sha: A *binary* SHA string. (20 characters long)_
       741│         """
    →  742│         assert len(sha) == 20
       743│         idx = ord(sha[:1])
       744│         if idx == 0:
       745│             start = 0
       746│         else:

❯ poetry add "git+https://github.com/dask/dask-kubernetes.git#dfeca2b"

Updating dependencies
Resolving dependencies... (2.4s)

Package operations: 0 installs, 1 update, 0 removals

  • Updating dask-kubernetes (0+untagged.571.g4ec02c5 4ec02c5 -> 0+untagged.571.g4ec02c5 dfeca2b
)

Writing lock file

[dbalabka]

That is another weird error. Installing the package from the scratch gets a different error. The repository contains all required metadata, however, poetry complaints that it is not:

❯ poetry add "git+https://github.com/dask/dask-kubernetes.git#dfeca2b"

Unable to determine package info for path: /home/dmitrybalabka/src/test-poetry/.venv/src/dask-kubernetes

Command ['/tmp/tmpipdj6o9y/.venv/bin/python', '-I', '-W', 'ignore', '-c', "import build\nimport build.env\nimport pyproject_hooks\n\nsource = '/home/dmitrybalabka/src/test-poetry/.venv/src/dask-kubernetes'\ndest = '/tmp/tmpipdj6o9y/dist'\n\nwith build.env.DefaultIsolatedEnv() as env:\n    builder = build.ProjectBuilder(\n        source_dir=source,\n        python_executable=env.python_executable,\n        runner=pyproject_hooks.quiet_subprocess_runner,\n    )\n    env.install(builder.build_system_requires)\n    env.install(builder.get_requires_for_build('wheel'))\n    builder.metadata_path(dest)\n"] errored with the following return code 1

Error output:
Traceback (most recent call last):
  File "<string>", line 9, in <module>
  File "/tmp/tmpipdj6o9y/.venv/lib/python3.10/site-packages/build/__init__.py", line 174, in __init__
    _validate_source_directory(source_dir)
  File "/tmp/tmpipdj6o9y/.venv/lib/python3.10/site-packages/build/__init__.py", line 77, in _validate_source_directory
    raise BuildException(msg)
build._exceptions.BuildException: Source /home/dmitrybalabka/src/test-poetry/.venv/src/dask-kubernetes does not appear to be a Python project: no pyproject.toml or setup.py

No fallback setup.py file was found to generate egg_info.

[jhaberstroh-sharethis]

I had the repo cloning working, but it just "randomly" started failing in this way. It even does it without the hash. Can someone with more familiarity with poetry put in code to catch this error and recommend this solution?

The behavior is highly variable after purging the repo trying to be updated from .venv OR deleting the entire venv & lock:

• When I use the 20 char SHA for every update, it works fine
• When I use 6 char SHA ever, I get one of the following in perpetuity for any attempt to update:

## Works randomly

## Throws assertion error 

❯ poetry update
Updating dependencies
Resolving dependencies... (1.1s)

Failed to clone [email protected]:-----/------.git at '03056d0', verify ref exists on remote.

Unable to determine package info for path: /Users/john/Library/Caches/pypoetry/virtualenvs/--------/src/--------

Command ['/var/folders/lc/kj4_ngks6w98yhrz7518cdb40000gp/T/tmpmz0p7l2s/.venv/bin/python', '-I', '-W', 'ignore', '-c', "import build\nimport build.env\nimport pyproject_hooks\n\nsource = '/Users/john/Library/Caches/pypoetry/virtualenvs/-------/src/-------'\ndest = '/var/folders/lc/kj4_ngks6w98yhrz7518cdb40000gp/T/tmpmz0p7l2s/dist'\n\nwith build.env.DefaultIsolatedEnv() as env:\n    builder = build.ProjectBuilder(\n        source_dir=source,\n        python_executable=env.python_executable,\n        runner=pyproject_hooks.quiet_subprocess_runner,\n    )\n    env.install(builder.build_system_requires)\n    env.install(builder.get_requires_for_build('wheel'))\n    builder.metadata_path(dest)\n"] errored with the following return code 1

Error output:
Traceback (most recent call last):
  File "<string>", line 9, in <module>
  File "/var/folders/lc/kj4_ngks6w98yhrz7518cdb40000gp/T/tmpmz0p7l2s/.venv/lib/python3.9/site-packages/build/__init__.py", line 174, in __init__
    _validate_source_directory(source_dir)
  File "/var/folders/lc/kj4_ngks6w98yhrz7518cdb40000gp/T/tmpmz0p7l2s/.venv/lib/python3.9/site-packages/build/__init__.py", line 77, in _validate_source_directory
    raise BuildException(msg)
build._exceptions.BuildException: Source /Users/john/Library/Caches/pypoetry/virtualenvs/--------/src/------- does not appear to be a Python project: no pyproject.toml or setup.py

No fallback setup.py file was found to generate egg_info.

This was seriously working without a hitch for months... what could have changed system-wide? I removed the entire venv and the lock, and the same thing is happening. I used 6-char SHA on Monday, and I just started getting the error today. The exact preceding incidents were:

• change to local dependency specification during development on dev branch
• change check out main, switching back to github dependency specification.
• run poetry update, requesting an older version of the dependency than the local

[aqeelat]

In my situation, the issue turned out to be a caching issue. For some reason, the package folder for the downloaded package source was empty. When I deleted it and ran poetry install, it worked fine. Mine was located at <venv>/src/<package_name>.

Possible reasons:

1. network issue when downloading the package
2. I deleted the package but the folder stayed there because of cosmic rays