Kyverno verify-flux-images missing digest bug

[schmidp]

Describe the bug

I am using the the sample kyverno policy to verify the flux-images:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-flux-images
spec:
  validationFailureAction: Audit
  background: false
  webhookTimeoutSeconds: 30
  failurePolicy: Fail
  rules:
    - name: verify-cosign-signature
      match:
        any:
        - resources:
            kinds:
              - Pod
      verifyImages:
        - imageReferences:
            - "ghcr.io/fluxcd/source-controller:*"
            - "ghcr.io/fluxcd/kustomize-controller:*"
            - "ghcr.io/fluxcd/helm-controller:*"
            - "ghcr.io/fluxcd/notification-controller:*"
            - "ghcr.io/fluxcd/image-reflector-controller:*"
            - "ghcr.io/fluxcd/image-automation-controller:*"
            - "docker.io/fluxcd/source-controller:*"
            - "docker.io/fluxcd/kustomize-controller:*"
            - "docker.io/fluxcd/helm-controller:*"
            - "docker.io/fluxcd/notification-controller:*"
            - "docker.io/fluxcd/image-reflector-controller:*"
            - "docker.io/fluxcd/image-automation-controller:*"
          mutateDigest: false
          attestors:
            - entries:
                - keyless:
                    subject: "https://github.com/fluxcd/*"
                    issuer: "https://token.actions.githubusercontent.com"
                    rekor:
                      url: https://rekor.sigstore.dev

But it seems like image-reflector-controller, image-automation-controller, notification-controller are missing their digest.

Steps to reproduce

1. Create kyverno policy as posted above
2. Provision flux with terraform:

resource "flux_bootstrap_git" "flux" {
  path = "clusters/${module.kube-or.cluster_name}"

  components_extra = [
    "image-automation-controller",
    "image-reflector-controller",
  ]

  depends_on = [module.kube-or]
}

Expected behavior

The policy is able to verify all images.

Screenshots and recordings

No response

OS / Distro

k3s on ubuntu 22.04

Flux version

flux: v2.0.0

Flux check

√ ~ % flux check ► checking prerequisites ✔ Kubernetes 1.27.3+k3s1 >=1.24.0-0 ► checking controllers ✔ helm-controller: deployment ready ► ghcr.io/fluxcd/helm-controller:v0.35.0 ✔ image-automation-controller: deployment ready ► ghcr.io/fluxcd/image-automation-controller:v0.35.0 ✔ image-reflector-controller: deployment ready ► ghcr.io/fluxcd/image-reflector-controller:v0.29.0 ✔ kustomize-controller: deployment ready ► ghcr.io/fluxcd/kustomize-controller:v1.0.0 ✔ notification-controller: deployment ready ► ghcr.io/fluxcd/notification-controller:v1.0.0 ✔ source-controller: deployment ready ► ghcr.io/fluxcd/source-controller:v1.0.0 ► checking crds ✔ alerts.notification.toolkit.fluxcd.io/v1beta2 ✔ buckets.source.toolkit.fluxcd.io/v1beta2 ✔ gitrepositories.source.toolkit.fluxcd.io/v1 ✔ helmcharts.source.toolkit.fluxcd.io/v1beta2 ✔ helmreleases.helm.toolkit.fluxcd.io/v2beta1 ✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2 ✔ imagepolicies.image.toolkit.fluxcd.io/v1beta2 ✔ imagerepositories.image.toolkit.fluxcd.io/v1beta2 ✔ imageupdateautomations.image.toolkit.fluxcd.io/v1beta1 ✔ kustomizations.kustomize.toolkit.fluxcd.io/v1 ✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2 ✔ providers.notification.toolkit.fluxcd.io/v1beta2 ✔ receivers.notification.toolkit.fluxcd.io/v1 ✔ all checks passed

Git provider

GitLab

Container Registry provider

GitLab

Additional context

No response

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[stefanprodan]

This is a bug in Kyverno, please open an issue on their repo. I guess Kyverno runs into rate limits and instead of retying it gives this bogus error message. You're not the first user reporting this.

All Flux images are signed with Cosign 2.0, you can verify them using the Cosign CLI:

cosign verify ghcr.io/fluxcd/source-controller:v1.0.0 \
--certificate-identity-regexp=https://github.com/fluxcd \
--certificate-oidc-issuer=https://token.actions.githubusercontent.com 

Verification for ghcr.io/fluxcd/source-controller:v1.0.0 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The code-signing certificate was verified using trusted certificate authority certificates

cosign verify ghcr.io/fluxcd/kustomize-controller:v1.0.0 \
--certificate-identity-regexp=https://github.com/fluxcd \
--certificate-oidc-issuer=https://token.actions.githubusercontent.com 

Verification for ghcr.io/fluxcd/kustomize-controller:v1.0.0 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The code-signing certificate was verified using trusted certificate authority certificates
 

cosign verify ghcr.io/fluxcd/notification-controller:v1.0.0 \
--certificate-identity-regexp=https://github.com/fluxcd \
--certificate-oidc-issuer=https://token.actions.githubusercontent.com 

Verification for ghcr.io/fluxcd/notification-controller:v1.0.0 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The code-signing certificate was verified using trusted certificate authority certificates
 
cosign verify ghcr.io/fluxcd/image-reflector-controller:v0.29.0 \
--certificate-identity-regexp=https://github.com/fluxcd \
--certificate-oidc-issuer=https://token.actions.githubusercontent.com 

Verification for ghcr.io/fluxcd/image-reflector-controller:v0.29.0 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The code-signing certificate was verified using trusted certificate authority certificates
 
cosign verify ghcr.io/fluxcd/image-automation-controller:v0.35.0 \
--certificate-identity-regexp=https://github.com/fluxcd \
--certificate-oidc-issuer=https://token.actions.githubusercontent.com 

Verification for ghcr.io/fluxcd/image-automation-controller:v0.35.0 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The code-signing certificate was verified using trusted certificate authority certificates
 
cosign verify ghcr.io/fluxcd/helm-controller:v0.35.0 \
--certificate-identity-regexp=https://github.com/fluxcd \
--certificate-oidc-issuer=https://token.actions.githubusercontent.com 

Verification for ghcr.io/fluxcd/helm-controller:v0.35.0 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The code-signing certificate was verified using trusted certificate authority certificates

I think it's time for the Flux team to find an alternative solution to Kyverno that knows how to handle rate limits and network failures, we should be looking into https://github.com/sigstore/policy-controller. cc @fluxcd/core-maintainers

[schmidp]

@stefanprodan ok I'll create an issue over at kyverno, closing this issue for now.

[stefanprodan]

Let's leave this open please, many users are running into it.