Poetry refuses to install package with correct hash

[MartinWallgren]

• [x] I am on the latest Poetry version.

• [x] I have searched the issues of this repo and believe that this is not a duplicate.

• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Debian Buster

• Poetry version: 1.1.9

• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/MartinWallgren/65ffceae6b597698602afdcdae927d7f

Issue

Poetry refuses to install a package even though the checksum is correct. Looking at the output it seems as if the cheksum stored in the lock file is md5 and the checksum used during installation is sha256.

Both sha256:3ae5020d5eddabcb57db9211e3f1a46ebafa28cb31cdeb4a497189041757bb7b and md5:75dbe554e7838a35e3a5836887cf9efc are valid checksums for this package according to our index (artifactory).

❯ poetry install
Installing dependencies from lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing axis-json-log-formatter (0.1.0): Failed

  RuntimeError

  Retrieved digest for link axis_json_log_formatter-0.1.0.tar.gz(sha256:3ae5020d5eddabcb57db9211e3f1a46ebafa28cb31cdeb4a497189041757bb7b) not in poetry.lock metadata ['md5:75dbe554e7838a35e3a5836887cf9efc']

  at ~/.poetry/lib/poetry/installation/chooser.py:115 in _get_links
      111│
      112│         if links and not selected_links:
      113│             raise RuntimeError(
      114│                 "Retrieved digest for link {}({}) not in poetry.lock metadata {}".format(
    → 115│                     link.filename, h, hashes
      116│                 )
      117│             )
      118│
      119│         return selected_links

[MartinWallgren]

I have tried to re-create the lock file without any difference in behavior.

[danygielow]

This seems to be a bug in poetry-core 1.0.5. With poetry-core 1.0.4 packages with md5 hashes are installed normally.

[orShap]

@danygielow looks like when trying to install poetry 1.1.8 in docker (with poetry-core 1.0.4 locked in poetry.lock file) it do installs 1.1.8 but somehow with poetry-core 1.0.5

#7 [ 4/10] RUN pip3 install pip==21.0.1 poetry==1.1.8
#7 sha256:8f42120a417b77040a77cf6cf1e2daeec7c2225b895246de3f5b5ecf08cc05e3
#7 2.256 Collecting pip==21.0.1
#7 2.276   Downloading pip-21.0.1-py3-none-any.whl (1.5 MB)
#7 2.560 Collecting poetry==1.1.8
#7 2.564   Downloading poetry-1.1.8-py2.py3-none-any.whl (173 kB)
#7 2.694 Collecting packaging<21.0,>=20.4
#7 2.698   Downloading packaging-20.9-py2.py3-none-any.whl (40 kB)
#7 2.730 Collecting crashtest<0.4.0,>=0.3.0
#7 2.734   Downloading crashtest-0.3.1-py3-none-any.whl (7.0 kB)
#7 2.847 Collecting requests<3.0,>=2.18
#7 2.852   Downloading requests-2.26.0-py2.py3-none-any.whl (62 kB)
#7 3.007 Collecting keyring<22.0.0,>=21.2.0
#7 3.011   Downloading keyring-21.8.0-py3-none-any.whl (32 kB)
#7 3.052 Collecting pexpect<5.0.0,>=4.7.0
#7 3.056   Downloading pexpect-4.8.0-py2.py3-none-any.whl (59 kB)
#7 3.127 Collecting poetry-core<1.1.0,>=1.0.4
#7 3.132   Downloading poetry_core-1.0.5-py2.py3-none-any.whl (424 kB)
...

1.1.8 poetry.lock file locks 1.0.4 https://github.com/python-poetry/poetry/blob/1.1.8/poetry.lock#L626

[danygielow]

@orShap The locked versions in poetry.lock are only used for poetry install. If you install poetry via pip the version constraints in pyproject.toml are used. In this case its >=1.0.4, <1.1 because it uses a tilde requirement. https://github.com/python-poetry/poetry/blob/1.1.8/pyproject.toml#L27

You could try: pip3 install pip==21.0.1 poetry==1.1.8 poetry-core==1.0.4

[hydra-zim]

according to pep-0503 , we should support all the hash functions in hashlib module in the Python standard library (currently md5, sha1, sha224, sha256, sha384, sha512)

[orShap]

@orShap The locked versions in poetry.lock are only used for poetry install. If you install poetry via pip the version constraints in pyproject.toml are used. In this case its >=1.0.4, <1.1 because it uses a tilde requirement. https://github.com/python-poetry/poetry/blob/1.1.8/pyproject.toml#L27

You could try: pip3 install pip==21.0.1 poetry==1.1.8 poetry-core==1.0.4

thanks! you're right

[sevaho]

Having a similar issue for private hosted pypi after upgrading to 1.1.9:

  • Updating wg-py-models (3.4.13 -> 3.4.15): Pending...
  • Updating wg-py-models (3.4.13 -> 3.4.15): Failed

  AttributeError

  'Link' object has no attribute 'is_absolute'

  at ~/.local/lib/python3.9/site-packages/poetry/core/packages/file_dependency.py:33 in __init__
       29│         self._path = path
       30│         self._base = base or Path.cwd()
       31│         self._full_path = path
       32│ 
    →  33│         if not self._path.is_absolute():
       34│             try:
       35│                 self._full_path = self._base.joinpath(self._path).resolve()
       36│             except FileNotFoundError:
       37│                 raise ValueError("Directory {} does not exist".format(self._path))

Could this be linked?

Can be mitigated by following the comment of @danygielow

@orShap The locked versions in poetry.lock are only used for poetry install. If you install poetry via pip the version constraints in pyproject.toml are used. In this case its >=1.0.4, <1.1 because it uses a tilde requirement. https://github.com/python-poetry/poetry/blob/1.1.8/pyproject.toml#L27

You could try: pip3 install pip==21.0.1 poetry==1.1.8 poetry-core==1.0.4

[mbelang]

Hitting the same issue but only for packages from our internal private Gemfury pypi repo. Poetry comes with standard CircleCI images and I do not have the control over the version that is installed there. Thanks to this issue, our entire stack is currently broken.

Update: The proposed fix didn't work for me, I had to do poetry self update 1.1.8

[hydra-zim]

This seems to be a bug in poetry-core 1.0.5. With poetry-core 1.0.4 packages with md5 hashes are installed normally.

It's not the bug in poetry-core 1.0.5, the problem always exists, but it happened that Package.clone omits files of Package, so the branch in executor.py#L673 is always passed.

After https://github.com/python-poetry/poetry-core/commit/3f718c55fcda63d9bd88b8fc612970c24fc9af25 , Package.clone use deepclone instead now, so the issue occurs

[bravefencermusashi]

Hello everyone similar problem here,

I have a dependency to a private package published through a private repository. That is what I have in my poetry.lock file:

my-package = [
    {file = "my-package-0.0.2-py3-none-any.whl", hash = "md5:3edd58d7d248531000f1e6671cf04c18"},
]

With poetry 1.1.8 everything is fine and the dependency can be installed with a poetry install, but with poetry 1.1.9 the installation (poetry install) fails with:

RuntimeError

 Invalid hash for my-package (0.0.2) using archive my-package-0.0.2-py3-none-any.whl

 at ~/.poetry/lib/poetry/installation/executor.py:619 in _download_link
     615│                     Path(archive.path) if isinstance(archive, Link) else archive,
     616│                 ).hash()
     617│             )
     618│             if archive_hash not in {f["hash"] for f in package.files}:
   → 619│                 raise RuntimeError(
     620│                     "Invalid hash for {} using archive {}".format(package, archive.name)
     621│                 )
     622│ 
     623│         return archive

I have cleared the poetry cache to make the tests There is only one version of the "my-package" which is 0.0.2 (the one I want to install) I have checked that the md5 is correct (lock vs real) my repo is pypiserver 1.4.2 I'm using python 3.8.8

I had to rollback to 1.1.8 to make it work

[adawalli]

If you want a way to downgrade the "jailed" version of poetry core in a one-liner

source $(dirname $(readlink -f $(which poetry)))/activate \
  && pip install poetry-core==1.0.4 \
  && deactivate

[moinerus]

rolled back to an old build image with poetry==1.1.8 would love to see this fixed asap so we can keep up with latest.

[pietrodn]

I can't reproduce the issue. All of the hashes in my poetry.lock are SHA256, not MD5. I've tried on existing virtualenvs as well as on fresh ones.

However some of my colleagues on the very same private Python index can reproduce the issue as for some reason they have MD5 hashes in the lockfile.

[pietrodn]

It seems that clearing the Poetry cache (and deleting the generated poetry.lock file) makes Poetry 1.1.9 use SHA256 instead of MD5, providing a workaround.

Try:

poetry cache clear . --all
rm poetry.lock
poetry install

[adawalli]

It seems that clearing the Poetry cache (and deleting the generated poetry.lock file) makes Poetry 1.1.9 use SHA256 instead of MD5, providing a workaround.

Try:

poetry cache clear . --all
rm poetry.lock
poetry install

beware: This will update all of the packages to the latest the resolver can resolve.

[qbedard]

It seems that clearing the Poetry cache (and deleting the generated poetry.lock file) makes Poetry 1.1.9 use SHA256 instead of MD5, providing a workaround.

Try:

poetry cache clear . --all
rm poetry.lock
poetry install

Nexus here, and this did not work for us. Followed these steps exactly and had the exact same result as before. Still getting md5 hashes. poetry 1.1.9, poetry-core 1.0.5.

[wcn00]

We (waveapps.com) have this problem too. All packages from our private repo fail to install with the same error as above. The above workaround does not work for me. However I can confirm that it is the packages using an md5 hash in the lock file are the ones that will not install.

[adawalli]

We (waveapps.com) have this problem too. All packages from our private repo fail to install with the same error as above. The above workaround does not work for me. However I can confirm that it is the packages using an md5 hash in the lock file are the ones that will not install.

Why doesn't the following work for you?

source $(dirname $(readlink -f $(which poetry)))/activate \
  && pip install poetry-core==1.0.4 \
  && deactivate

[qbedard]

I should clarify that pinning poetry-core to 1.0.4 does work for me. @wcn00 Have you tried this yet (per @adawalli)?

[wcn00]

We (waveapps.com) have this problem too. All packages from our private repo fail to install with the same error as above. The above workaround does not work for me. However I can confirm that it is the packages using an md5 hash in the lock file are the ones that will not install.

Why doesn't the following work for you?

source $(dirname $(readlink -f $(which poetry)))/activate \
  && pip install poetry-core==1.0.4 \
  && deactivate

That will work. However doing this means that we have to package and build dozens of packages to get started. This problem just started happening TODAY. It means that every build that pulls one of our custom python libs (most of them) will fail.

[adawalli]

Yes, the same problem just started affecting us today as well. Thankfully, I ship poetry in a base image to all of my other images, so the fix was in one place. The poetry installer doesn't allow you to lock in core unfortunately. I haven't tried pip installing poetry, but I really like to "jail" poetry, which is why I really like the installer script....

[qbedard]

That will work. However doing this means that we have to package and build dozens of packages to get started. This problem just started happening TODAY. It means that every build that pulls one of our custom python libs (most of them) will fail.

Yeah. Feeling that pain right now.

[qbedard]

Should this be an issue on poetry-core?

[odavid]

I believe It is indeed the change, however, the real issue is the fact poetry code assumes the checksums are always sha256 and does not respect md5 checksums. The change in poetry-core made poetry start validating checksums for link artifacts, which is a valid behavior, so I think the bug should be fixed in poetry code.

On Mon, 20 Sep 2021 at 22:36 Tim Bedard @.***> wrote:

Not sure why yet, but pretty sure it's this: https://github.com/python-poetry/poetry-core/pull/193/files

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/python-poetry/poetry/issues/4523#issuecomment-923234324, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGSOAMZNHLHCNF5U7OBREDUC6EMDANCNFSM5ELFRVHQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[mstandley-tempus]

It seems that clearing the Poetry cache (and deleting the generated poetry.lock file) makes Poetry 1.1.9 use SHA256 instead of MD5, providing a workaround. Try:

poetry cache clear . --all
rm poetry.lock
poetry install

Nexus here, and this did not work for us. Followed these steps exactly and had the exact same result as before. Still getting md5 hashes. poetry 1.1.9, poetry-core 1.0.5.

For other Nexus users: clearing the cache and regenerating the lock file won't work, because Nexus currently returns MD5s (even though poetry 1.1.9 wants it to return SHA256s).

You can follow along with this Nexus problem here: https://issues.sonatype.org/browse/NEXUS-24127

[MartinWallgren]

It seems that clearing the Poetry cache (and deleting the generated poetry.lock file) makes Poetry 1.1.9 use SHA256 instead of MD5, providing a workaround.

Try:

poetry cache clear . --all
rm poetry.lock
poetry install

To highlight, this is a workaround at best. It's not a viable permanent solution to force projects to regenerate the lock file.

Hopefully one of the connected PRs can get some attention python-poetry/poetry-core/pull/206 looks like a quick fix.

[MartinWallgren]

I just noticed that PR python-poetry/poetry-core/pull/206 is already merged. Hopefully it solves the problem.

[odavid]

I just noticed that PR python-poetry/poetry-core/pull/206 is already merged. Hopefully it solves the problem.

I feel it will be only part of the solution, since in the executor.py code, the validation logic is using hardcoded sha256

[sevaho]

For people who are using pypiserver you can update the --hash-algo without any side effects (at least not on my side).

From the docs:

  --hash-algo ALGO
    Any `hashlib` available algo used as fragments on package links.
    Set one of (0, no, off, false) to disabled it (default: md5).

So I've updated it to:

--hash-algo sha256

Full command:

pypi-server -p 8080 -a update -P /etc/nginx/.htpasswd --hash-algo sha256 --log-file /var/log/pypi/pypiserver.log /var/www/pypi

Then increase the version of your private package and install it in your app. It should update the md5 sum in poetry.lock to a sha256 sum.

[alexef]

For the record, for people using Nexus, the temporary workaround is to pin it down to:

poetry==1.1.8
poetry-core==1.0.4