Push signatures first

[mattmoor]

Generally whenever distroless publishes new images, we see a rash of CI failures (e.g. cosigned e2e tests) because the tagged images aren't signed.

Looking through, distroless is also unfortunately signing tags (bad!): https://github.com/GoogleContainerTools/distroless/blob/3fe389d8dd2021f9f17f5de863bdfe3b9792ceec/cloudbuild_cosign.sh#L12-L48

This is part of why ko publishes SBOMs before publishing the images, although here it's tricky because a lack of integration in the build tooling.

cc @loosebazooka @imjasonh @jonjohnsonjr @dlorenc

[imjasonh]

The cosign version is also old:

https://github.com/GoogleContainerTools/distroless/blob/1a3e90ffe07519e8586f37dc956f93aeb87e09b1/cloudbuild.yaml#L53

I'm not sure it's possible to sign the platform-specific images before they're pushed (I'm also not sure it's useful), but I think it might be possible to sign the multi-platform manifest list we all know and love before pushing it.

That's done here:

https://github.com/GoogleContainerTools/distroless/blob/1a3e90ffe07519e8586f37dc956f93aeb87e09b1/cloudbuild_docker.sh#L15

Between docker manifest create and docker manifest push you could take the digest of that manifest with docker manifest inspect $_image | sha256sum -, sign that, then push the manifest.

[imjasonh]

docker manifest inspect helpfully inserts a trailing newline, which isn't there when you push it. So:

  _digest=$(printf "%s" "$(docker manifest inspect ${_image})" | sha256sum -)                                                                                                                             
  cosign sign ${_image}@${_digest}
  docker manifest push ${_image}

The next trick is going to be having a GCB step that has both docker and cosign available. Today that's two steps, one that has docker, and one that has cosign -- there's also one that installs and invokes jq.

[loosebazooka]

Sgtm. What I really want to do is just stage every build and then crane copy (or cosign copy) over the images to prod gcr. But that can be for another day

[mattmoor]

Staging also sounds good, but we should sign the digests wherever we sign them 😅

[loosebazooka]

this is fixed by rules_oci