Allow configuring whether image is replaced with SHA digest

[wojciechka]

The policy-controller mutating webhook is patching images in PodSpec to replace tags with digests - for example docker.io/library/telegraf:1.22 in Deployment is expanded to index.docker.io/library/telegraf@sha256:05b59d689b7812cfc31a3a2eb70fe18d3adea74a1d572fbf1a91294e091359e7.

I understand the rationale behind it, but it would be great to allow some configuration to prevent this behavior either at workload, namespace, or global level.

Applying manifests via CD (or probably other reconciliation tools as well) will cause the tool to go into indefinite loop of trying to update the image. This happened in an environment with ArgoCD v2.2.11 - where ArgoCD was reporting a difference between expected and live manifest and trying to reconcile it, only to be updated by the webhook.

This is fine when having control over the manifests - it forces pinning to specific SHA digest.

However, we sometimes use CD for deploying workloads we do not fully control or want to control - such as Helm charts or other third-party manifests where the image may be specified as tag, and this will cause CD to go into an infinite loop/conflict with policy-controller.

It would be great if this would be configurable, preferably not globally as the behavior may be forcing good practices in some cases.

Confirmed on version 0.3.0

[slimm609]

How would you validate the signature of the container? It resolves the tag to an immutable reference after validation. If it leaves the tag, there is no way to validate the image during admission and would have to revalidate on each node when pulling the image.

Pinning to a sha is because that’s the immutable reference which was signed.

[wojciechka]

How would you validate the signature of the container? It resolves the tag to an immutable reference after validation. If it leaves the tag, there is no way to validate the image during admission and would have to revalidate on each node when pulling the image.

I was wondering if the reason is to make sure the image that was validated ends up the one being run - because otherwise there's always a race between what is validated and what gets pulled when the Pod gets created.

Would it be possible to update the image at the Pod or ReplicaSet level instead? Or make that behavior configurable.

Pod level patching would be slower but running more replicas of policy-controller-webhook should still make it usable.

Perhaps caching could also help here as well, so over time policy-controller-webhook pods use the cached tag -> digest mapping and apply the changes faster.

Pinning to a sha is because that’s the immutable reference which was signed.

I understand and I like the idea. It's just that it's hard to configure CD systems (ArgoCD in our case) to at the same time apply an image but ignore any changes that other controllers are applying.

Changing child objects instead of the original object is much easier as CD systems would not consider this to be a drift. This is especially hard to coordinate in the cases where manifests are not managed by teams that own the clusters.

[hectorj2f]

@wojciechka From a security point of view, I won't go this way and force everyone to use sha instead of tags. On the other hand, this could be possibly handled via a configuration property in sigstore/policy-controler configMap. @wojciechka Do you feel you could help adding that change to the controller ?

[wojciechka]

@wojciechka From a security point of view, I won't go this way and force everyone to use sha instead of tags. On the other hand, this could be possibly handled via a configuration property in sigstore/policy-controler configMap. @wojciechka Do you feel you could help adding that change to the controller ?

Yes, I'd like to give it a try on my own.

I will get myself familiar with developing policy-controller and look into this after KubeCon.