investigate kyverno post-deploy diff

[cppforlife]

Describe the problem/challenge you have

After first deploy with below yaml, try changing audit => enforce and see diff on the next deploy.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-ro-rootfs
  annotations:
    policies.kyverno.io/title: Require Read-Only Root Filesystem
    policies.kyverno.io/category: Best Practices
    policies.kyverno.io/severity: medium
    policies.kyverno.io/subject: Pod
    policies.kyverno.io/description: >-
      A read-only root file system helps to enforce an immutable infrastructure strategy; 
      the container only needs to write on the mounted volume that persists the state. 
      An immutable root filesystem can also prevent malicious binaries from writing to the 
      host system.
spec:
  validationFailureAction: audit
  background: true
  rules:
  - name: validate-readOnlyRootFilesystem
    match:
      resources:
        kinds:
        - Pod
    validate:
      message: "Root filesystem must be read-only."
      pattern:
        spec:
          containers:
          - securityContext:
              readOnlyRootFilesystem: true

how can we improve this flow?

Describe the solution you'd like [A clear and concise description of what you want to happen. If applicable a visual representation of the UX (ex: new CLI argument name, the behavior expected).]

Anything else you would like to add: [Additional information that will assist in solving the issue.]

---

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible" 👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.

[100mik]

On replicating this issue. In the diff we can see two objects on the rules list that were not added by us:

 75     -   - exclude:
 76     -       resources: {}
 77     -     generate:
 78     -       clone: {}
 79     -     match:
 80     -       resources:
 81     -         kinds:
 82     -         - DaemonSet
 83     -         - Deployment
 84     -         - Job
 85     -         - StatefulSet
 86     -     mutate: {}
 87     -     name: autogen-validate-readOnlyRootFilesystem
 88     -     validate:
 89     -       message: Root filesystem must be read-only.
 90     -       pattern:
 91     -         spec:
 92     -           template:
 93     -             spec:
 94     -               containers:
 95     -               - securityContext:
 96     -                   readOnlyRootFilesystem: true
 97     -   - exclude:
 98     -       resources: {}
 99     -     generate:
100     -       clone: {}
101     -     match:
102     -       resources:
103     -         kinds:
104     -         - CronJob
105     -     mutate: {}
106     -     name: autogen-cronjob-validate-readOnlyRootFilesystem
107     -     validate:
108     -       message: Root filesystem must be read-only.
109     -       pattern:
110     -         spec:
111     -           jobTemplate:
112     -             spec:
113     -               template:
114     -                 spec:
115     -                   containers:
116     -                   - securityContext:
117     -                       readOnlyRootFilesystem: true
118     -   validationFailureAction: audit
     69 +   validationFailureAction: enforce
119, 70   

Kyverno seems to generate additional rules when we define a rule for a Pod so that it applies to pod templates in Deployment, DaemonSet, StatefulSet, Job, and CronJob resources. This behaviour is documented here. Both of the two autogenerated names have the autogen- prefix. A potential solution for this would be to exclude items in a list as per a user-defined ytt rule before diffing. Something similar to what we have in place here. We could allow the user to deal with cases like this in a declarative manner.

There seems to be a noisy diff due to the resource being restructured and the addition of default values by Kyverno:

57, 56     rules:
 58     -   - exclude:
 59     -       resources: {}
 60     -     generate:
 61     -       clone: {}
 62     -     match:
     57 +   - match:
 63, 58         resources:

In addition, Kyverno adds an annotation listing the resources it created autogen rules for:

annotations:
  4     -     pod-policies.kyverno.io/autogen-controllers: DaemonSet,Deployment,Job,StatefulSet,CronJob