ko
ko copied to clipboard
Signing built images
Users can sign images produced with ko publish
using tools like cosign
.
For example:
$ cosign sign -key cosign.key $(ko publish ./)
ko resolve
produces potentially many images, which makes this a bit harder. You could ko resolve
then scan the resulting YAML for ko
-built image references and sign all of those with some bash magic, but 🤮 .
Would it be useful to have a ko resolve --sign cosign.key
flag that used the provided key to sign all images built during ko resolve
?
ko publish --sign cosign.key
could also be a convenience alias for, effectively, cosign sign $(ko publish)
, which wouldn't require users to have cosign
installed.
@dlorenc good idea? bad idea?
@dlorenc good idea? bad idea?
Great idea!
This is where I really want a sidechannel for build outputs (my stupid "layoutput" thing).
+1 to native integration here.
Generally my preference would be to bias towards the ephemeral key route and use the same ephemeral key to sign all of the images produced (we should probably also start to think about incorporating a variety of opinionated attributes based on things we know about the build).
re: many images
something like mink
produced ~25 images at its peak, for ~5 architectures, so 1 key vs ~150 keys (25*5 + 25) is probably a better way to go 🤣
This PR includes a technique I have been using to incorporate keyless signing into some ko-like tools: https://github.com/sigstore/cosign/pull/647
Basically it only kicks in when COSIGN_EXPERIMENTAL=true
for now, which is similar to how we've gated certain features with GGCR_EXPERIMENTAL_*
, and this feels like an extremely lightweight way to integrate things.
We sacrifice some amount of configurability, but my inclination here would be to probably adjust how cosign
itself works to use envconfig
as a way to support a standard scheme for env-var fallback for configuration (no cobra, so no viper).
+1 to KO_EXPERIMENTAL_SOMETHING
to iterate on this. How do we imagine this looking when it graduates from an experiment? We don't have a good example of doing that in GGCR as far as I know.
Ideally we'd end in a state where things are signed by default using ambient tokens if available, which can be disabled with a flag or config. Do we want to support signing with explicit keys? Or just leave that for cosign?
I think the first order of business will be to get the distroless images signed against Fulcio.
For configuring things, I think trying to standardize on env vars in cosign upstream would be my preference (for portability across tools).
For opting out (beyond experiment), I think some .ko.yaml syntax is called for, we can introduce this to ease folks into it too.
Another step between “experiment guarded” and “opt-out” to encourage folks to start signing will be to always verify, but make verification non-fatal (print warnings).
I also think we probably want some sort of refresh token flow for users before we move beyond experimental.
sorry this ended up sort of stream of consciousness.
Ok, full keyboard so here's sort of what I'm thinking in terms of phasing this in.
- No change unless TBD
*EXPERIMENT*
env var is set, at which point base images must be signed, and published images are signed.
(once the default base images are signed against Fulcio)
- Start verifying base images all the time, but verification is only fatal when TBD
*EXPERIMENT*
flag is set. When it isn't set we emit a warning. We still only sign things when it is set. We will need to sort out a way to opt-out of verification for each base image (likely in.ko.yaml
?).
(once keyless signing is no longer "experimental" in Sigstore)
- Start signing output images whenever ambient OIDC auth is available. Provide a way to require this (e.g. release envs), or opt-out entirely (e.g. dev environments avoiding 3LOs).
(once we have a clear opt-out path for base images)
- Start to make verification of base images fatal by default.
As with --platform=all
, we should never require signing by default (e.g. this would slow dev loops with 3LOs, and break using ko
in PRs which won't have OIDC).
If we find out that being more aggressive signing things in dev is important (e.g. back-pressure from cosigned
rejecting things), then we can always revisit this, but hopefully by then we have some sort of OIDC refresh token flow.
Starting to poke through code, I think the most natural integration point for signing is likely to implement a new publisher.Interface
that composes with another publisher similar to the caching version:
// Wrap publisher in a memoizing publisher implementation.
return publish.NewCaching(innerPublisher)
}
If we have a signing publisher, we can accumulate / pass through images to an inner publisher, and then sign all of the accumulated images in Close()
before closing the inner publisher. I think this would mean we use a single key to sign the entire set of images we publish 🤞
As I'm writing this I realize that cosign
likely doesn't have functions that take a v1.Image
(really build.Result
) and return the v1.Image
containing the signature, so we can't leverage the inner publisher to write the signature, which means the only publisher this should compose with right now is the default publisher. It would be cool to write such a function in cosign
because it would allow us to emit signed images in tarballs and OCI layouts as well.
So it looks like currently:
return cli.Sign().Exec(context.Background(), n.digests.List())
... generates a new cert for every image, and send the user through a separate 3LO for each one, so on big projects (e.g. knative.dev/serving
) this is sort of prohibitive.
cc @dlorenc
Have we thought through what new CLI flags we'd need?
Regardless of how cosign-as-an-sdk evolves, cosign sign
has quite a lot of flags -- are we going to need all of these in every ko
surface?
ko resolve -f config/ \
--fulcio-url https://staging.fulcio.not-sigstore.dev \
-a foo=bar \
--oidc-issuer=some.other-issuer.example \
...
Maybe we should wait for the [EXPERIMENTAL] stuff to become un-experimental -- sad as it would be to have to wait for the nice things -- before adding all those cosign
flags to ko
?
Maybe we should wait for the [EXPERIMENTAL] stuff to become un-experimental -- sad as it would be to have to wait for the nice things -- before adding all those
cosign
flags toko
?
I don't really think experimental will change much in the flag surface either way.
Picking this back up, since I think we're a lot closer to having this be possible than we were six months ago.
The concerns I think we all have are roughly threefold:
- scary [EXPERIMENTAL] warnings around keyless and Rekor -- this is going to go away soonish, when Sigstore goes GA.
- dependency graph explosion: cosign depends on a bunch of stuff we don't need. This is a known issue we're now trying to tackle better upstream.
- CLI surface explosion:
cosign sign
takes a bunch of flags, which we don't necessarily want to just blindly copy intoko build
et al.
To solve (3) I'd like to propose we start by only having ko
sign things if it detects it can do so keylessly, using ambient OIDC credentials -- e.g., running in a GitHub Actions workflow with idtoken:write
pushing to GHCR, or on GCB pushing to GCR/AR. Since this assumes an unattended flow, it will also only upload to Rekor if it detects the repo is public, and not ask for confirmation.
This eliminates the need for flags to configure a key, including references to keys in KMS systems, and security key flows. Instead of having flags for --fulcio-url
, --oidc-issuer
, etc. flags, these will come from standard environment variables, which we expect only to be used when building against your own Sigstore infrastructure.
Signing will be disabled by default, behind a new flag, --sign
/-s
.
If we're happy with the added dependencies required to get this working, we're pretty close to having the user-signing OIDC flow too, using sigstore/sigstore's pkg/oauthflow
. This will will pop up a browser to go through OAuth before keylessly signing and uploading to Rekor -- if the image repo is private, we'll ask for confirmation before uploading to Rekor.
btw, after shelving https://github.com/google/ko/pull/433 for adding ~400kloc of dependencies, we had about half that many lines sneak in through https://github.com/google/ko/pull/718 anyway, so I think the dependency ship may have sailed ⛵. Not that we shouldn't keep working to get cosign's dependencies down, but maybe it's worth just biting the bullet and getting signing in before cutting out those deps.
Once we have image signing in place we should also have ko
sign the SBOMs it produces, or attach them as signed attestations instead of unsigned SBOMs.
Some more considerations I think we'll need to...consider:
- We probably don't want to pop up an OAuth flow for each image we build and sign, despite wanting to keylessly sign each image. We can cache the certificate, but it's only valid for 10 minutes, and the time between the first image and the last image in a
ko resolve
may be longer than 10 minutes. - Lots of times, a
ko resolve
on N images won't actually produce N new images; should we sign them all each time they're a part of ako resolve
, even if there's no new data being signed? That seems potentially spammy. We can try deduping? Maybe? Latest signature wins and overwrites the oldest? - We shouldn't publish to Rekor until after the image is successfully pushed (in case we don't have permission to push to that registry), so we'll need to keylessly sign, hold on to that cert(s), then push the image and signatures, then publish to Rekor for all the images we built+signed+pushed, using the correct cert(s) for each image we pushed. Should we publish in a batch at the end of
ko resolve
? Probably not, since Fulcio's certs may be expired by the time we finishresolve
ing. If we publish to Rekor ASAP duringko resolve
, we'll probably want to have some summary of what we did printed at the end, or else all the useful Rekor verification info is interleaved in the (already pretty noisy) build logs.
I figured we'd have something like git commit -s
to trigger signing. We could also use .ko.yaml
to default -s
on like I do with gitsign
for example.
I figured we'd have something like
git commit -s
to trigger signing. We could also use.ko.yaml
to default-s
on like I do withgitsign
for example.
Is that effectively "ko
shells out to cosign
" then? If someone configures a cosign sign
that doesn't do anything, ko
would be oblivious. And we'd probably want to make sure the ko image contains cosign. Bleh.
gitsign has to deal with a lot of these issues too, fwiw, to prevent OAuth popup spam while always providing a valid unexpired cert (https://github.com/sigstore/gitsign/pull/75). I'd kind of rather write that in Go, even if it's painful.
gitsign has to live in Git's constraints, ko is much more free to do better.
I think that with -s
we'd also want to encode the SBOMs as attestations, so I suspect that'd be really cumbersome via shelling out (certainly no way to avoid multiple 3LOs).
I think that with
-s
we'd also want to encode the SBOMs as attestations, so I suspect that'd be really cumbersome via shelling out (certainly no way to avoid multiple 3LOs).
Yeah. I think we're talking about the same thing. -s
should sign all the images it builds, and attach SBOMs as signed attestations in addition to (instead of?) unsigned SBOMs.
The problems in https://github.com/google/ko/issues/357#issuecomment-1164767219 roughly concern how to avoid multiple unnecessary 3LOs, since that will really ruin the experience of using ko
anywhere outside of CI.
gitsign
currently avoids it by running a daemon that it connects to over a socket to get cached certs, but that's because how gitsign
is invoked is up to Git and not gitsign -- it's invoked once for every commit that needs to be signed and Rekor-published, and we're invoked once to potentially build N images, some of which may not be new and worthy of signing or Rekording.
Some more considerations I think we'll need to...consider:
- We probably don't want to pop up an OAuth flow for each image we build and sign, despite wanting to keylessly sign each image. We can cache the certificate, but it's only valid for 10 minutes, and the time between the first image and the last image in a
ko resolve
may be longer than 10 minutes.
Nope, this was dumb. We should just build all the images we're going to build (1 for ko build
, N for ko resolve
) and do the 3LO once and sign all those images with the resulting Fulcio certificate. If any build fails, don't sign anything.
- Lots of times, a
ko resolve
on N images won't actually produce N new images; should we sign them all each time they're a part of ako resolve
, even if there's no new data being signed? That seems potentially spammy. We can try deduping? Maybe? Latest signature wins and overwrites the oldest?
I still don't have a good answer to this.
- We shouldn't publish to Rekor until after the image is successfully pushed (in case we don't have permission to push to that registry), so we'll need to keylessly sign, hold on to that cert(s), then push the image and signatures, then publish to Rekor for all the images we built+signed+pushed, using the correct cert(s) for each image we pushed. Should we publish in a batch at the end of
ko resolve
? Probably not, since Fulcio's certs may be expired by the time we finishresolve
ing. If we publish to Rekor ASAP duringko resolve
, we'll probably want to have some summary of what we did printed at the end, or else all the useful Rekor verification info is interleaved in the (already pretty noisy) build logs.
I think doing one signing pass after all the builds complete means this isn't an issue. It shouldn't take 10m just to push the signatures.
I still don't have a good answer to this.
I might have an okay answer to this.
ko build
and ko resolve
should have a --sign
/-s
flag, but ko apply|create|run
shouldn't. The latter are intended for hot-inner-loop development where signing doesn't make sense; the former are generally for publishing stuff to other folks, where you should sign it.
@imjasonh - may be -sign/s
with --sign-key
wouldn't be better to start with? We can then start adding other OIDC signing methods once the experimental flags on them are removed?
OIDC signing for cosign is no longer experimental, after the recent Sigstore GA.
I still think I'd like to focus on keyless signing in ko
, since that's going to end up being the lowest-friction way to generate signed release artifacts, especially using GitHub Actions OIDC. We'll undoubtedly need to support keyful signing too, but my hope is that's rarer.
We've also been talking about plumbing sufficient environment variables into cosign
to configure custom sigstore instances. The idea is to enable a pattern like minikube's env
setup:
eval $(configure-sigstore-env)
cosign sign foo
Where the former emits a set of:
FOO=bar
BAZ=blah
I think we should aim to align with that model for configuring ko
so that it can compose with tooling the same way cosign
does.
Perhaps we should have a way to configure keyful via the env too?
OIDC signing for cosign is no longer experimental, after the recent Sigstore GA. I still need to use
COSIGN_EXPERIMENTAL
for doing OIDC
I love the keyless signing but then wondering how we can configure keyless signing w/o browser, typically to have integration with CI/automation. I see lots of --odic*
options to cosign but wondering will we be able to configure github/google oAuth via those options ( if so excuse my ignorance here). I see this https://docs.sigstore.dev/cosign/openid_signing#oauth-flows but not sure that helps CI/automation cases as there will be manual intervention
Perhaps we should have a way to configure keyful via the env too? IMHO that should be, as may CI tools set values using env vars
I also faced an dependencies issue as detailed here https://github.com/sigstore/cosign/issues/2477
I love the keyless signing but then wondering how we can configure keyless signing w/o browser, typically to have integration with CI/automation. I see lots of
--odic*
options to cosign but wondering will we be able to configure github/google oAuth via those options ( if so excuse my ignorance here). I see this https://docs.sigstore.dev/cosign/openid_signing#oauth-flows but not sure that helps CI/automation cases as there will be manual intervention
GitHub OIDC and Google and Amazon Workload Identity OIDC are all pretty straightforward to detect and use automatically, without flags.
The oidc
flags for cosign sign
for example mostly duplicate those that would be configured by the env vars Matt described.
The oidc flags for cosign sign` for example mostly duplicate those that would be configured by the env vars Matt described.
That should be better, we should wait for it IMHO
@imjasonh when trying the https://docs.sigstore.dev/cosign/openid_signing#oauth-flows I still see we get prompted to open browser or other kind of manual intervention. Not sure how they play with automation/CI workflow , I mean just by providing --oidc*
parameters.
When you run cosign sign
inside GitHub Actions with OIDC enabled (id-token: write
), cosign
will automatically pick up the credentials from the environment and not require a browser.
For example:
jobs:
sign:
runs-on: ubuntu-latest
permissions:
packages: write
id-token: write
steps:
- uses: sigstore/cosign-installer@main
- run: cosign sign ghcr.io/${{github.repository}}
This will sign the image as the GitHub Actions workload identity. You can see what that looks like in Rekor here: https://rekor.tlog.dev/?logIndex=7576978
To do this, cosign
checks for the presence of a ACTIONS_ID_TOKEN_REQUEST_URL
env var, which is set when the workflow is run with the id-token:write
permission. If set, requests that URL with another GitHub-provided token. The response is an OIDC token that cosign
can pass to Fulcio to get a short-lived signing cert, which it uses to sign the image and log to Rekor.
All of this should already be abstracted by sign.SignCmd
, and ko
would ideally do a similar thing using a future better-factored Go API.
@imjasonh have you tried using GitHub OIDC outside of GH Actions similar to how we can do GCP --identity-token
signing ? I checked the GH Docs but not finding any clue on how to get the id-token
from GH PAT