Annotate policies with Kubernetes compatibility version

[chipzoller]

Per prior agreement, add new annotations to policies which list the tested version of Kubernetes and Kyverno.

[JimBugwadia]

Hi Chip, my understanding was we would start with something like kyverno.io/kubernetes-version: 1.20.+ to allow the policy authors to indicate which version the policy is written for, and we would automate tests for the 3 prior versions. This way, as Kubernetes objects evolve, policy authors have a way to manage policies for newer and older versions.

Does this match what you have in mind?

[chipzoller]

I think we need to be clear. The annotation can indicate one of two things:

• The minimum version of Kubernetes on which the policy, as written, will function as intended.
• The version of Kubernetes on which the policy is confirmed to work (i.e., the version used to test the policy by the contributor).

Which are we envisioning here? They aren't necessarily the same.

[JimBugwadia]

Agreed, they are not the same, and perhaps we need both.

I feel that #1 is critical. The versions of Kubernetes on which the policy, as written, will function as intended. Note that I removed minimum as this needs to be a range of versions for which the policy will operate.

We are seeing this as a real problem with customers, as policies need to evolve as Kubernetes resources evolve.

The version on which the policy is confirmed to work is also interesting. Ideally its always tested for all supported versions, and I see no reason why we cannot automate this for the policy library (once we have the intended versions.)

[chipzoller]

I also think having a range of versions of Kubernetes would be valuable, but here are the three challenges I see are before us.

1. The value of this annotation has to factor in Kyverno versions as well. Take the Block Pod Exec by Namespace Label example. This policy will only work beginning in 1.4.2 as we state in the minversion annotation. That version of Kyverno has a minimum supported version of Kubernetes as well. Assuming that's 1.16 as we state in the matrix, does the value of this new annotation become 1.16-1.22? Obviously Kubernetes allowed Pod exec operations long before 1.16, so does the annotation refer to the Kubernetes capabilities on which the policy is predicated (i.e., operations, API versions, and schema elements) or does it refer to the technology needed to fully realize the intent of the policy, a union of Kyverno and Kubernetes? How about the Require Labels policy? Kubernetes has had labels for a long time, and I'll bet this policy was probably one of the first use cases enabled by Kyverno. We'd have to go back quite a ways to figure out where the first versions of Kubernetes and Kyverno intersected to enable this policy as it's written. Let's say we find it and it happens to be 1.6 (picking an old release), so the value of the annotation is reflected as 1.6-1.22. In order for that to actually work on 1.6, one would have to get a really old version of Kyverno (let's assume at least one version was compatible with 1.6). This is now outside the N-2 support schedule of Kyverno. Many folks are still running old versions of Kubernetes in production, even PaaS operators! Are we 1) endorsing the use of an ancient and potentially less secure of Kyverno by advertising this? 2) signing up to support users in these situations? 3) also indicating a way a user can actually implement the policy on such a version by helping them locate such an old version of Kyverno?
2. The necessity of timely policy annotation updates after Kubernetes releases. Let's say 1.21 is the current release. We have 100 policies in the samples repo which have, to-date, accurate values for their kyverno.io/kubernetes-version annotations. Fast forward three months and 1.22 is now available. A policy whose annotation states, for example, 1.16-1.21 might also be valid through 1.22. We're now at 1.22 and the policy in question has not been "certified" for 1.22. A user finds this policy on the policy repo/website and finds the max stated version of K8s is 1.21. They're running 1.22 and deduce it will not work for them. They now potentially pass on the policy because of skepticism that it'll work in their environment. So we're committing to the continual upkeep of this annotation across N number of policies or we risk providing stale information which misleads users. This is related to point number three.
3. The onus of responsibility to continually perform re-certification; the minimum requirement for acceptance. We have few maintainers and contributors as of now and most of those efforts are focused on high value tasks like code contributions and documentation. Where is the onus of maintaining not just the policies themselves but the librarian task of performing upkeep of this annotation? It's much more of a time commitment if we want it to be meaningful. And how will that be done? Automated testing through some framework sounds good, but who's writing those? When will they be written? Would we just have to manually go through each and every one approximately every three months to keep them fresh? Again, it's a commitment we need to think about.

I understand the ostensible usefulness of something like this might be to indicate in the Unique Ingress Host policy, for example, that the API call, as written presently, is only going to be applicable starting in Kubernetes 1.19, but that's an easy case of identification. If we want to be consistent--and I believe we should be--there are more implications we need to think about and whether this will be sustainable over time.

[chipzoller]

Somewhat separately (but still related) on your comment:

We are seeing this as a real problem with customers, as policies need to evolve as Kubernetes resources evolve.

Taking the Unique Ingress Host example again. Kubernetes version N is released which bumps the API version to v2. The kyverno.io/kubernetes-version on the existing policy reflects 1.19-N. Do we now basically duplicate the policy with the only difference being the API version assuming nothing in the schema (with respect to this policy, as written) has changed? That seems wasteful. Do we put a comment in the existing policy to acknowledge starting in which version of Kubernetes the API version needs to be changed? Or do we just leave it be and let users figure it out on their own? Do we have a rule where we can update the existing policy to state v2 after Kubernetes 1.19 is officially unsupported (while updating the value of kyverno.io/kubernetes-version)?

[JimBugwadia]

Yes, we will need a Kyverno and Kubernetes version for each policy. Something like:

annotations:
  kyverno.io/kyverno-version: 1.4.+
  kyverno.io/kubernetes-version: 1.20.+

The testing and labels can be limited to the 3 prior supported minor versions of each, yielding a 3x3 test matrix. So, in your examples no need to go back in time and list all supported K8s versions, etc.

We can document and follow some simple conventions which make it easy to search and replace lower versions as we update to new versions.

Additionally, we can start versioning (tagging) the policy library. This will allow going back to a prior version of the policy library for folks who want to know what versions of Kyverno and Kubernetes were previously tested.

[chipzoller]

Yes, we will need a Kyverno and Kubernetes version for each policy. Something like:

annotations:
  kyverno.io/kyverno-version: 1.4.+
  kyverno.io/kubernetes-version: 1.20.+

I'm not a fan of using a plus notation in this context. Think of scenario number two in my previous reply. We shouldn't assume a policy on the current release (of either Kyverno or Kubernetes) will work for any future releases.

[chipzoller]

Although these annotations have been adopted by the new PSS policy revamp plus other new policies from this point on, let's leave this open as a reminder to go back through the policy library and apply them to all others.