Attestation error: "Verification failed: subject does not match distribution name"

[arahlin]

I'm getting an error about invalid attestations trying to publish my project to PyPI. I use a pretty typical workflow: cibuildwheel to create wheels, and upload-artifact/download-artifact actions to collect all the wheels into one directory for this pypi upload action.

It looks like something is mangling a wheel name in the attestation subject in the process, as the upload fails on one of the wheels with this error:

Uploading spt3g-1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
WARNING  Error during upload. Retry with the --verbose option for more details. 
ERROR    HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/        
         Invalid attestations supplied during upload: Could not verify the      
         uploaded artifact using the included attestation: Verification failed: 
         subject does not match distribution name:                              
         spt3g-1.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl !=
         spt3g-1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl   

This is my workflow, and this is the failed action run.

I've disabled attestations for now, but am reporting this so that it can be resolved for future releases.

[webknjaz]

@woodruffw has anything changed in PyPI regarding sorting the wheel tags?

@bdraco just pointed me at the same problem roughly 20 minutes ago: https://github.com/aio-libs/frozenlist/actions/runs/15404234130/job/43344457100#step:9:823.

I haven't noticed any obvious changes in Warehouse that would be related. Although... could it be the recent deps bump?

[webknjaz]

Could https://github.com/pypi/warehouse/pull/18140 be at fault? (though https://github.com/pypi/warehouse/pull/18220)

[webknjaz]

It's actually bumped in https://github.com/pypa/gh-action-pypi-publish/pull/359 too but isn't yet released.

I imagine cutting a new release might solve it.

As a workaround, it should be possible to pin to the last commit or the unstable branch.

[arahlin]

Following a few links down through #359, I think pypa/auditwheel#583 might be the actual culprit, so presumably a new auditwheel release (or perhaps an update to cibuildwheel) would also solve this.

[webknjaz]

Ouch.. Though, this would mean that PyPI also sorts them but differently, maybe?

cc @di @henryiii for visibility

[woodruffw]

Thanks for the ping. That Warehouse PR seems like the likely culprit, although I don't understand the "why" of it yet -- that bump should make wheel tag comparisons more permissive, not less. I'll be able to diagnose more when I'm in front of a computer.

[webknjaz]

@woodruffw would it matter that I haven't yet tagged the action with the bumped dep?

[henryiii]

We had to adapt for this in cibuildwheel's test suite in https://github.com/pypa/cibuildwheel/pull/2426. It's an auditwheel change. Comparisons in theory should be done using packaging's normalized name tuple. Order doesn't matter in the spec.

These will suddenly become a lot more common when we release cibuildwheel 3.0 (very soon).

[webknjaz]

Thanks for another data point! I'll wait for the discussion here overnight and get back to this tomorrow.

[woodruffw]

Okay, I think I've tracked this down at least on the PyPI side -- the problem is that pypi-attestations removed the "ultranormalization" step, but the post-ultranormalization comparison wasn't changed.

Specifically, these are the offending lines:

https://github.com/trailofbits/pypi-attestations/blob/1ba4cb1d0925778f49c52479e7b9a5097f988397/src/pypi_attestations/_impl.py#L286-L302

In the previous version both subject.name and dist.name were ultranormalized, but now they're whatever the user passes in (so long as it can be parsed as a dist filename). So the string comparison fails where it previously succeeded due to ultranormalization.

The fix here is to not do a string comparison at all here, and compare the parsed results as @henryiii says. At least, until the entire ecosystem starts emitting properly sorted wheel tags (which will naturally take a very long time).

TL;DR: This is surfacing a regression in pypi-publish that I'll fix now. In the mean time, PyPI should also probably revert that pypi-attestations bump to minimize breakage.

CC @miketheman for a potential revert, if you're up 🙂 -- I can send the revert PR if it makes things easier ^

Order doesn't matter in the spec.

The spec is ambiguous here, IMO -- here's what it says:

To allow for compact filenames of bdists that work with more than one compatibility tag triple, each tag in a filename can instead be a ‘.’-separated, sorted, set of tags.

Ref: https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/#compressed-tag-sets

(The basic ambiguity is whether the spec is using "set" as a shorthand for "collection" or to mean Python set; if it's the latter, then it's a contradiction in terms that probably needs an errata.)

[woodruffw]

https://github.com/trailofbits/pypi-attestations/pull/127 will fix this upstream.

[henryiii]

I think we should hold off on a final 3.0 release of cibuildwheel until this is resolved. I'll watch this issue.

[henryiii]

, each tag in a filename can instead be a ‘.’-separated, sorted, set of tags.

I don't think that's ambiguous, comparing these as sorted strings is fine. You just have to have the same sorting on both strings! In general, that means other you must always sort both strings yourself, or you can compare via a set. (edit: ah, you mean sorted(set(..)) while a literal translation sounds like set(stored(...)). I don't think that's the problem here, though, it wasn't randomly ordered, it changed the way it was sorted.

[facutuesca]

The fix for this has been merged and included in a new release of pypi-attestations: v0.0.27

I've opened a PR in warehouse to bump the version: https://github.com/pypi/warehouse/pull/18221

[miketheman]

I've opened a PR in warehouse to bump the version: pypi/warehouse#18221

This has been merged and deployed.

[woodruffw]

I don't think that's the problem here, though, it wasn't randomly ordered, it changed the way it was sorted.

I might not be following what you mean, but I think these are effectively the same shape of problem: if the spec is insufficiently clear about what "sorted" means here, then we probably need some kind of errata clarifying it to prevent these kinds of confusions in the future.

Or in other words: if my interpretation of the intent behind the tags spec is correct, then wheel filenames should be comparable as strings without having to do any additional normalization or re-sorting. It's just the case that nobody is currently enforcing that, and that the ecosystem as a whole is permissive of non-sorted compressed tags in wheel filenames 🙂

(You're right that the problem here isn't exactly that, though -- the immediate problem here is a plain old bug caused by me accidentally only half-fixing the normalization behavior in pypi-attestations 😅.)

[henryiii]

Yes, I think you are right. Might be worth mentioning that not everyone sorts these correctly so comparisons should be made on re-sorted tags / sets.

So we are clear for a cibuildwheel release? Still ironing out iOS stuff but otherwise I think it's ready.

[woodruffw]

Triage: I think this has been addressed, at least in an intermediate sense, by the pypi-attestations release and subsequent deployment to Warehouse.

[henryiii]

We just released cibuildwheel 3.0, so I expect you'll find out pretty soon if it's not. :)

[webknjaz]

v1.13.0 bumps to the latest version of pypi-attestations and with that, I think there's no more outstanding action items remaining. Closing.