python-libusb1 icon indicating copy to clipboard operation
python-libusb1 copied to clipboard
verified publisher icon vpelletier

Fix Release Signatures on PyPI (404 Not Found)

Open maltfield opened this issue 1 year ago • 5 comments

Describe the bug The cryptographic signature of the libusb1 module on PyPI has gone missing (404 Not Found)

To Reproduce

Steps to reproduce the behavior (following the instructions found here):

  1. Fetch the URL of the libusb1-3.1.0 .whl file from the pypi.org simple API
  2. Append .asc to the URL of the .whl file's URL and attempt to download it
  3. Get 404 error

Example execution

user@disp9050:~$ curl -s https://pypi.org/simple/libusb1/ | grep -i libusb1-3.1.0-py3-none-any.whl
<a href="https://files.pythonhosted.org/packages/85/5c/9169aea7690df382b677d9f725accc3ec864849c5ab49991e3823a942392/libusb1-3.1.0-py3-none-any.whl#sha256=9d9f16e2c199cab91f48ead585d3f5ec7e8e4be428a25ddfed22abf786fa9b3a" data-dist-info-metadata="sha256=55c3b663f491b900e4f7ac9dc3d85172342188660fe2f6bc2a41e9de40500811" data-core-metadata="sha256=55c3b663f491b900e4f7ac9dc3d85172342188660fe2f6bc2a41e9de40500811">libusb1-3.1.0-py3-none-any.whl</a><br />
user@disp9050:~$ 

user@disp9050:~$ wget https://files.pythonhosted.org/packages/85/5c/9169aea7690df382b677d9f725accc3ec864849c5ab49991e3823a942392/libusb1-3.1.0-py3-none-any.whl.asc
--2024-02-26 19:31:54--  https://files.pythonhosted.org/packages/85/5c/9169aea7690df382b677d9f725accc3ec864849c5ab49991e3823a942392/libusb1-3.1.0-py3-none-any.whl.asc
Resolving files.pythonhosted.org (files.pythonhosted.org)... 151.101.164.223, 2a04:4e42:27::223
Connecting to files.pythonhosted.org (files.pythonhosted.org)|151.101.164.223|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2024-02-26 19:31:57 ERROR 404: Not Found.

user@disp9050:~$

Note that the file is available; just the signature is not.

user@disp9050:~$ wget https://files.pythonhosted.org/packages/85/5c/9169aea7690df382b677d9f725accc3ec864849c5ab49991e3823a942392/libusb1-3.1.0-py3-none-any.whl
--2024-02-26 19:32:25--  https://files.pythonhosted.org/packages/85/5c/9169aea7690df382b677d9f725accc3ec864849c5ab49991e3823a942392/libusb1-3.1.0-py3-none-any.whl
Resolving files.pythonhosted.org (files.pythonhosted.org)... 151.101.164.223, 2a04:4e42:27::223
Connecting to files.pythonhosted.org (files.pythonhosted.org)|151.101.164.223|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 62368 (61K) [application/octet-stream]
Saving to: ‘libusb1-3.1.0-py3-none-any.whl’

libusb1-3.1.0-py3-n 100%[===================>]  60.91K   175KB/s    in 0.3s    

2024-02-26 19:32:27 (175 KB/s) - ‘libusb1-3.1.0-py3-none-any.whl’ saved [62368/62368]

user@disp9050:~$

Expected behavior

I should be able to download both the libusb1 payload and its cryptographic signature as before (and my builds shouldn't be breaking due to this bug).

maltfield avatar Feb 27 '24 00:02 maltfield

Note that this issue does appear to be specific to libusb1 and not an upstream issue with PyPI.

I confirmed that the signature is still available for another project (borgbackup):

user@disp9050:~$ curl -s https://pypi.org/simple/borgbackup/ | grep -i borgbackup-1.1.13.tar.gz
<a href="https://files.pythonhosted.org/packages/97/68/27d96a12f54894223ad6676ce4d215ad61771e3e723580f3ee6e609e17b7/borgbackup-1.1.13.tar.gz#sha256=164a8666a61071ce2fa6c60627c7646f12e3a8e74cd38f046be72f5ea91b3821" >borgbackup-1.1.13.tar.gz</a><br />
user@disp9050:~$ 

user@disp9050:~$ wget https://files.pythonhosted.org/packages/97/68/27d96a12f54894223ad6676ce4d215ad61771e3e723580f3ee6e609e17b7/borgbackup-1.1.13.tar.gz
--2024-02-25 22:49:21--  https://files.pythonhosted.org/packages/97/68/27d96a12f54894223ad6676ce4d215ad61771e3e723580f3ee6e609e17b7/borgbackup-1.1.13.tar.gz
Resolving files.pythonhosted.org (files.pythonhosted.org)... 151.101.124.223, 2a04:4e42:1e::223
Connecting to files.pythonhosted.org (files.pythonhosted.org)|151.101.124.223|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3754457 (3.6M) [application/x-tar]
Saving to: ‘borgbackup-1.1.13.tar.gz’

borgbackup-1.1.13.t 100%[===================>]   3.58M  42.6KB/s    in 99s     

2024-02-25 22:51:13 (37.0 KB/s) - ‘borgbackup-1.1.13.tar.gz’ saved [3754457/3754457]

user@disp9050:~$ 

user@disp9050:~$ wget https://files.pythonhosted.org/packages/97/68/27d96a12f54894223ad6676ce4d215ad61771e3e723580f3ee6e609e17b7/borgbackup-1.1.13.tar.gz.asc
--2024-02-25 22:51:18--  https://files.pythonhosted.org/packages/97/68/27d96a12f54894223ad6676ce4d215ad61771e3e723580f3ee6e609e17b7/borgbackup-1.1.13.tar.gz.asc
Resolving files.pythonhosted.org (files.pythonhosted.org)... 151.101.124.223, 2a04:4e42:1e::223
Connecting to files.pythonhosted.org (files.pythonhosted.org)|151.101.124.223|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 862 [application/octet-stream]
Saving to: ‘borgbackup-1.1.13.tar.gz.asc’

borgbackup-1.1.13.t 100%[===================>]     862  --.-KB/s    in 0.007s  

2024-02-25 22:51:19 (115 KB/s) - ‘borgbackup-1.1.13.tar.gz.asc’ saved [862/862]

user@disp9050:~$ 

user@disp9050:~$ gpg --verify borgbackup-1.1.13.tar.gz.asc
gpg: assuming signed data in 'borgbackup-1.1.13.tar.gz'
gpg: Signature made Sat 06 Jun 2020 05:37:32 PM -05
gpg:                using RSA key 2F81AFFBAB04E11FE8EE65D4243ACFA951F78E01
gpg:                issuer "[email protected]"
gpg: Good signature from "Thomas Waldmann <[email protected]>" [unknown]
gpg:                 aka "Thomas Waldmann <[email protected]>" [unknown]
gpg:                 aka "Thomas Waldmann <[email protected]>" [unknown]
gpg:                 aka "Thomas Waldmann <[email protected]>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 6D5B EF9A DD20 7580 5747  B70F 9F88 FB52 FAF7 B393
     Subkey fingerprint: 2F81 AFFB AB04 E11F E8EE  65D4 243A CFA9 51F7 8E01
user@disp9050:~$

maltfield avatar Feb 27 '24 00:02 maltfield

Thank you for this detailed bug report.

Note that this issue does appear to be specific to libusb1 and not an upstream issue with PyPI.

Indeed. Looks like it is even specific to the latest release:

$ curl --silent --request HEAD --write-out '%{http_code} %{url}\n' $(curl -s https://pypi.org/simple/libusb1/ | grep -i py3-none-any.whl | sed 's/.*href="\([^#]*\).*/\1.asc/')
200 https://files.pythonhosted.org/packages/fa/8e/7f3fdabe77d2b03e8eec31874bc4ecdb73b4f085698ccbc75b2b105b6dfd/libusb1-1.9-py3-none-any.whl.asc
200 https://files.pythonhosted.org/packages/3c/4b/0c8b566d21f8920019237cc9f919e0aeeb43aa350819ea4bcdd2788f39a3/libusb1-1.9.1-py3-none-any.whl.asc
200 https://files.pythonhosted.org/packages/14/bd/29515a44719569aa0942551571d89539a1c0633eb78a6d695b462145827f/libusb1-1.9.2-py3-none-any.whl.asc
200 https://files.pythonhosted.org/packages/b6/f7/be261cd16470bb732137288da2ea09fe983935325e47f986c42ca098eec2/libusb1-1.9.3-py3-none-any.whl.asc
200 https://files.pythonhosted.org/packages/75/02/d609f752af9cd0365479875a28d2f4ad7adfa9b1aa6772445bcc6884fe15/libusb1-1.10.1-py3-none-any.whl.asc
200 https://files.pythonhosted.org/packages/08/57/f07415f47e9fcd4034c73f0b5a72b752c6a82089f74c9b3d304c9a7e3a49/libusb1-2.0.0-py3-none-any.whl.asc
200 https://files.pythonhosted.org/packages/d2/43/86ee846c6c2ac858645eca239030247fae4f5d764177ce4fb9bedb05e41a/libusb1-2.0.1-py3-none-any.whl.asc
200 https://files.pythonhosted.org/packages/e0/a0/bfe18a27c5c8e8922bdd7f1ac67c10da907b95c00c3baa92e0dd463d02b6/libusb1-3.0.0-py3-none-any.whl.asc
404 https://files.pythonhosted.org/packages/85/5c/9169aea7690df382b677d9f725accc3ec864849c5ab49991e3823a942392/libusb1-3.1.0-py3-none-any.whl.asc

IIRC it is not possible to add files to an existing release, so I most likely have to release a new version.

While I guess it does not help much now, I see I did generate those signatures when I released and they do match the twine upload command pattern setup.sh outputs at the end, so I have no idea how they did not get uploaded while the rest of the release was:

$ release_prefix=dist/libusb1-3.1.0
$ ls -l ${release_prefix}-*.whl{,.asc} ${release_prefix}.tar.gz{,.asc}
-rw-r--r-- 1 vincent vincent  62368 29 oct.  03:09 dist/libusb1-3.1.0-py3-none-any.whl
-rw-r--r-- 1 vincent vincent    833 29 oct.  03:11 dist/libusb1-3.1.0-py3-none-any.whl.asc
-rw-r--r-- 1 vincent vincent 127838 29 oct.  03:09 dist/libusb1-3.1.0-py3-none-win32.whl
-rw-r--r-- 1 vincent vincent    833 29 oct.  03:11 dist/libusb1-3.1.0-py3-none-win32.whl.asc
-rw-r--r-- 1 vincent vincent 140380 29 oct.  03:09 dist/libusb1-3.1.0-py3-none-win_amd64.whl
-rw-r--r-- 1 vincent vincent    833 29 oct.  03:11 dist/libusb1-3.1.0-py3-none-win_amd64.whl.asc
-rw-r--r-- 1 vincent vincent  83013 29 oct.  03:09 dist/libusb1-3.1.0.tar.gz
-rw-r--r-- 1 vincent vincent    833 29 oct.  03:11 dist/libusb1-3.1.0.tar.gz.asc

vpelletier avatar Feb 27 '24 13:02 vpelletier

In addition to attempting a new release, can you also please add all the release files as a release on GitHub?

That would provide a redundant method for downstream consumers of python-libusb1 to be able to safely obtain this dependency, even if there's issues on PyPI's website.

maltfield avatar Feb 27 '24 14:02 maltfield

Good idea. I created the 3.0.0 and 3.1.0 releases.

Also, as libusb1 released 1.0.27 since I released 3.1.0 I intend to port the features it brings to python-libusb1 before doing a new release, so it is going to need more time.

vpelletier avatar Feb 28 '24 01:02 vpelletier

I created the 3.0.0 and 3.1.0 releases.

Great, that unblocks us for now so we can safely build our app again. Thanks :)

  • https://github.com/vpelletier/python-libusb1/releases/tag/3.1.0

maltfield avatar Feb 28 '24 04:02 maltfield

Uploading 3.2.0 to pypi, I notice a twine warning:

WARNING  One or more packages has an associated PGP signature; these will be silently ignored by the index

Which led me to a related warning message in the source, which led me to a twine bug report about removing GPG support which led me to the pypi notice about them dropping GPG support because not enough people use it.

So I'm closing this bug report, and will continue creating releases here with corresponding signatures.

I have no idea if/how you can automatically discover new releases, nor how you can cross-reference a github-hosted signature of a pypi-discovered new version.

vpelletier avatar Dec 29 '24 12:12 vpelletier

Sad to see the pip devs choosing to make their platform insecure :(

Good thing that the actual python team does sign their releases; it's just pip that's horribly insecure and must be avoided now.

I will use your PGP signature that you provide on your releases downstream. Thank you for continuing to provide a means for us downstream devs to obtain your software securely <3

maltfield avatar Jan 02 '25 03:01 maltfield