Ensure unsigned images never get published

[castrojo]

We got a few rando reports of image upgrade failures, p5 investigated and it was a temporary outage from sigstore:

Error: signing [ghcr.io/ublue-os/kinoite-nvidia@sha256:2219734f69fc517aac7f744c2f9af0f5fc54796b8a866a9ee686cd0f73b72edf]: signing digest: Post "https://rekor.sigstore.dev/api/v1/log/entries": POST https://rekor.sigstore.dev/api/v1/log/entries giving up after 4 attempt(s)

Rebuilds fixed the issue:

j0rge: should we have a step that verifies the signatures on each build maybe? I don't recall this happening before?

p5: A verify step would only be useful if we ran it before pushing the image (or tagging the images with the final tag). Currently signing happens after pushes, so will block nothing

KyleGospo: I think we need a cleanup action that removes these failed builds sigstore going down shouldn't leave us with an unsigned image though at least it's a self-solving problem

[p5]

Something we could consider is to push the images to GHCR with a temporary tag (or no tag?), sign the images with cosign using the digest as the selector, then set the final tags via Skopeo. This would mean that the final "production" tags are only added onto the image after the image signing step was successful.

1. Push images to GHCR with temporary/no tag
2. Tell cosign to sign the image with the digest
3. Add the required tags to the image

[castrojo]

This happened today with Bluefin LTS. James thinks we should add a check to the workflows so that we do not tag the image until after we sign and confirm it.