Admission controllers for validating & mutating the resources

[nolar]

Add some way to valudate and mutate the resources immediately when they are created, i.e. before the Kubernetes API (kubectl) call finishes.

import kopf

@kopf.on.validate('zalando.org', 'v1', 'kopfexamples')
def check_body(spec, **_):
    if spec.get('field') is None:
        raise kopf.ValidationError("spec.field is required.")

@kopf.on.validate('zalando.org', 'v1', 'kopfexamples', field='spec.items')
def check_items(value, **_):
    if not all(item.startswith('item') for item in value):
        raise kopf.ValidationError("All items must start with 'item' text.")

If any of the validation handlers failed, the object creation should also fail.

Read more:

• https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/

This implies that the operator reacts to the API calls, so the deployment pattern should be extended with the kind: Service objects, and the port listening on the pod. Related: #18 for the health-checks, with also require a listening port.

TODO: Need more info on how the admission controllers work (beside creating a kind: ValidatingWebhookConfiguration object).

[nolar]

Example use-case: Automatically injecting the last-update timestamp when an object is created/updated, to implement the TTL-like cleanup of the stale objects (not updated for few weeks).

• https://twitter.com/try_except_/status/1127484958258487297
• https://twitter.com/try_except_/status/1127494195881619456
• https://twitter.com/nolar/status/1127504800122396673

[nolar]

Taking the discussion from #49: @rosscdh asked:

I was thinking something like either another decorator or appending a spec_validator to the current decorator which if present would then validate just pre to being injected into the create|delete|etc|_fn(spec) tho problems cloud arise if there are version changes to an existing spec whos schema then gets updated.. maybe spec versioning? needs a bit of thought.

Separate question, how to ensure that the spec provided by the CRD is the expected spec.. is there a prescribed manner? i.e in Go the specs are validated and contolled by structs, but in kopf? with dynamic python?

Generally speaking, Kopf does not do this right now. You can define a basic spec in a CRD, and it will be ensured by K8s itself (without reaching the operators, and without even accepting that object). More sophisticated checks are only possible via the validation hooks, and therefore #55 (it will be difficult because of how these hooks work).

As a simple workaround, you can decorate your handler functions with any validation framework, and fail them if the spec or body or meta kwargs are not as wanted. The custom resource will be in the cluster anyway (accepted by K8s), but with no reaction to it after the failure. You can also post k8s events on it (stored for 1hr), or write its status for clarity (stored forever).

I don't think that bringing the spec-validation logic into Kopf itself is needed at this point and this way, as it can be done in Kopf-based operators independently of Kopf as a framework. Kopf, in the first place, is a framework to marshal K8s events to Python code (and back), and support any domains/business logic. People can have their own preferences on validation libraries. The only thing Kopf can do here, is to be friendly (compatible, integrable) to these validation libraries as much as possible.

That approach, however, can be documented as an example of the validation and put to the Kopf's docs (or maybe better a blog post somewhere?).

[rosscdh]

Perfect thanks for taking the time to explain that (of course the CRD is the specification, I do beg your pardon for not thinking of that).!

[jkroepke]

Field validation for CRD fields in kubernetes is in beta state.

See https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#validation

[krish7919]

A related note: can we extend kopf to act as both a validating admissions webhook, as well as custom controller?

This should be in the scope of this project?

EDIT:

I see in https://github.com/zalando-incubator/kopf/issues/240 that this was tentatively planned for early 2020, any update? :)

[nolar]

@krish7919 Yes, having validating & mutating handlers is intended for Kopf — i.e., the feature is in Kopf's scope. Beside, it is one of three remaining major features left to implement to reach the original vision: daemons & timers; admission hooks; cross-resource communication.

But I would not promise anything regarding plans or dates now — we change the plans and priorities on the go (work-related, not Kopf-related), and I've just finished my personal mega-struggle with daemons & timers (in a release candidate yet).

Once 0.27 (non-RC) is released, I will take a look on how difficult or easy it will be to implement the admission hooks now (with the new source code layout) — and so for many other requests/issues in the backlog. Some of them definitely became much more reachable in the past half-year.

[Frankkkkk]

Hi @nolar , I'd be interested in working on this feature (admission controller) because of a need we have at work. However, I see that you're planning a major code rework, so I'm wondering if this would be a good option to do it now ? If that's, would you have some guidelines on how to do it "the clean way" on the code base ? Kopf's code is way more complicated than I thought.

I was thinking of:

• creating an engines/admission.py with a TCP server. The server is "launched" the same way as the probing one
• on each AdmissionReview request, parse the body of the CR and call the corresponding kopf.on.validate function from the registry ?
• return the response object with allowed = True iff the function returned True ?

However, there are still some fundamental questions about who creates the ValidatingWebhookConfiguration object:

• If the user does it, then Kopf's dev mode wouldn't work nicely: you'd have to change the resource to point it to your dev machine (and it's implications: forwarding the ports to the cluster, etc..)
• If Kopf's creates the object, then IMHO it's cleaner but what happens when there are multiple operators (and only "master" one) running on the cluster ? The master would have to change the object when it's his time to process. A svc would not be mandatory (i.e. we could put the pod's address)
• We need to take into account use cases like one pod with 2 containers (each container being kopf run a.py and kopf run b.py): there would need to be a separate TCP port for each of these operators (as they'd use the same network ns).

Well, that's actually not an easy one :stuck_out_tongue:

Thanks and cheers !

[vaidik]

It will be awesome to get this feature. I am working around right now by patching the object I wish to mutate. It'll be better if I mutate though.

[eshepelyuk]

Also interested in this feature. Kopf usage looks really promising for developing admission controllers.

[nolar]

@Frankkkkk Yes, it is not an easy task for sure :-) Too many uncertainties. Before I can answer them, I need to play with admission hooks, so that I would have hands-on feeling of what they are — not just from the docs.

---

The TCP ports and listening would not be a big problem — it can be done the same way as health probing with a CLI option like --liveness=http://:8080/healthz (https://kopf.readthedocs.io/en/latest/probing/) — and make it the developer's responsibility to separate containers. Or maybe run it as one operator with kopf run a.py b.py — depends on the use-case.

The dev-mode is the biggest challenge. One of the main Kopf's promises is the human friendliness, which includes running in IDEs locally. One way would be to sacrifice that promise — temporarily or permanently — and implement the hooks when running in the cluster only. When and if possible, port forwarding can be added later.

---

Another way would be to inject some dev-mode port forwarding.

On the one hand, as far as I understand this this article, it can be implemented via running a CLI tool in a pod, and forwarding the TCP traffic via stderr/stdout.

On the other hand, there are K8s API endpoints for proxying: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#-strong-proxy-operations-pod-v1-core-strong- — not very descriptive, so some playing is needed again.

---

ValidatingWebhookConfiguration/MutatingWebhookConfiguration resources can be created and removed dynamically on operator startup & shutdown: either one per run (but: what will happen if the operator dies?), or named after a local operator id (persistent hashing of a hostname+username or something) and reused on restarts, or via CLI option names.

---

Also, beside doing al this magic with ports & hooks, Kopf's own DSL should be clarified. There are 2 types of hooks: the mutation and the validation hooks. Both should be supported by Kopf.

And it is a thing to clarify, how does it interact with Python's handler functions (i.e. marshalling K8s to Python and back to K8s) — returned results, arbitrary exceptions, Kopf's TemporaryError/PermanentError, etc.

---

So, yes, a lot of uncertainties, which will require a lot of effort. I might take this task as the next big thing after the planned 0.28 & 0.29 — see milestones — both are implemented in parts here & there in the branches (see my fork), but not merged and assembled yet.

As I said in another comment somewhere, the past few months were very intense (oh corona!), and I had no time for Kopf: barely finished the daemons& timers (started Nov'2019). I am right now in a process of adjusting that, and hopefully will have clarity on the further development in a few weeks.