warehouse icon indicating copy to clipboard operation
warehouse copied to clipboard

Store attestations for PEP740

Open DarkaMaul opened this issue 1 year ago • 1 comments

Context

This PR follows #16063 and https://github.com/pypi/warehouse/pull/15952 and build on them. After verifying attestations, this PR introduces their storage and retrieval when needed.

Main changes

At upload time

  • After attestations have been processed, they are not discarded but stored in using the storage service.
  • A Provenance file is generated and stored with the release file

At download time

  • When requesting a release using the simple API (JSON/HTML), if attestations exist for the release :
    • a new JSON key provenance with the sha256 of the provenance file is set
    • or, a data attribute data-provenance is set with the same value

Implementations choices

  • Attestations are stored based on their hash in buckets:
attestation_sha256_digest = ...
attestation_path =    "/'.join(
    attestation_sha256_digest[:2],
    attestation_sha256_digest[2:4],
    attestation_sha256_digest[4:],
    f"{release_path}.attestation",
)
  • Provenance files are stored along the release file to comply with PEP740

TODOs

  • [ ] Improve and fix the tests
  • [ ] documentation on the newly created subpackage

/cc @woodruffw @facutuesca

DarkaMaul avatar Jul 18 '24 13:07 DarkaMaul

Let put this one on hold until https://github.com/trailofbits/pypi-attestations/pull/36 is merged.

DarkaMaul avatar Jul 26 '24 16:07 DarkaMaul

This is looking really good, nice work @DarkaMaul!

To confirm my understanding of the flow here:

  1. A package is published using TP and contains one (currently just one) attestation in the attestations POST field;
  2. We parse and verify that attestation against the TP identity that published the package file
  3. We persist the attestation as its own file and content-address it, referencing it from the new Attestation model
  4. We generate the provenance object from the attestation and store it.

Did I get this right?

Exactly

DarkaMaul avatar Aug 15 '24 13:08 DarkaMaul