Reconciling resources

[klarose]

Expected Behavior

An important part of kubernetes is the idea of eventual consistency. If a document is applied to the cluster, the state it describes may not take effect immediately, or the cluster may deviate occasionally from it. I want to be able to write operators which take this principle to heart.

In order to do this, I think that we need to be able to add something akin to reconciliation loops in https://github.com/operator-framework/operator-sdk.

For example, if I create a resource, say MyDeployment. MyDeployment contains allows me to create k8s Deployment objects simplified for my specific Docker image.

If something changes the Deployment generated by MyDeployment, I want to be notified of that change so that I may place it back into the desired state.

Actual Behavior

From reading the documentation, there doesn't appear to be any way to be notified when an 'owned' resource deviates from the desired state. It would be nice if we could do this in a similar decorator-style fashion as the other kopf.on module decorators. I haven't thought enough about what a good API would look like here. I think it would be necessarily different from the go operator-sdk method, since it handles everything in one loop, wheareas kopf splits things up for you..

Other stuff

Thanks for doing this! I've implemented a few off-the-cuff operators in python. What you've done here would have simplified them dramatically.

[nolar]

@klarose First of all, thanks for your warm words! This keep me motivated and drives the project :-)

Also thanks for the detailed suggestion. This in indeed an important feature to have, and we discussed it from different points in other contexts too.

A very important part of a reconciliation loop is knowing the state of the children. In the current implementation, Kopf only knows the state of the operated resource. In some cases, the children can be other k8s resources, but it is also possible to have some third-party API children objects: orchestrate from K8s via CRDs, but execute/store in an external environment via APIs.

Kopf itself cannot manage all of such states. But it can help.

---

Currently, a reconciliation loop can be implemented via @kopf.on.resume + @kopf.on.create handlers (the same function with 2 decorators). The handler should start a thread or a task with while True and polling the state of the children/external resources every N seconds.

An example of this is in examples/10/builtins. Also see #122.

import asyncio
import kopf

@kopf.on.resume('zalando.org', 'v1', 'KopfExample')
@kopf.on.create('zalando.org', 'v1', 'KopfExample')
async def start_monitoring(**kwargs):
    asyncio.create_task(monitoring(**kwargs))

async def monitoring(**kwargs):
    while True:
        check_it()
        update_it()
        await asyncio.sleep(60)

That works, but can be not very native. And it is not event-driven. And it is wordy (see examples/10).

---

Another way how this could be implemented soon (not now) is via timing decorators (maybe #19):

@kopf.on.timer('zalando.org', 'v1', 'kopfexamples', interval=60)
def reconcile_it(**kwargs):
    check_it()
    update_it()

Would that be more convenient?

I can get to that after the current topic (stability and resilience) is finished. The foundation is already implemented (since the time when this issue was created), and this feature can be added.

---

There is another idea that I have in mind — cross-resource handlers. I don't know how the syntax could look like yet, but the idea is that the handler is declared for a child resource (Deployment), but is also attached to the "related" resource (MyDeployment); or vice versa: @kopf.on.child is declared on a parent resource (MyDeployment), but is restricted to the specific children (Deployment) — same as @kopf.on.field restricts to specific fields.

The idea, in English, is to trigger a handler and to do things on MyDeployment when something happens on its children Deployment (or e.g. Pod).

The "relation" can be defined in multiple ways. One is via metadata.ownerReferences (parent-children). Another one is via selectors (specifically, by labels). These two are actively used in K8s itself. Also, some arbitrary selectors can be used. All other non-selected resources of that kind will be ignored. For that, a declarative filtering system is being implemented in #58 #45 #98.

It is not yet defined in which context should a cross-resource handler be executed: a child or a related resource, and how the second one is passed to the function (i.e. the values of body, spec, and other kwargs, and the function result handling).

[klarose]

I think your second suggestion certainly goes a long way to helping out. It's nice to not need to define an explicit task to handle things.

That said, the final idea you mention seems like exactly what I want -- an explicit link between parent and child, so that if the child changes the parent is notified.

In the go operator SDK, the link between 'primary' and 'secondary' types is explicit.

As far as I can tell, the sdk handles the watching/notification part of things for you, but it's up to the implementer to actually fetch the resources on a change. If we want to simplify that here, you're right: we'll need to figure out how to intuitively pass the child resource to the handler.

One idea: what if the decorator could give each child resource a name. Then, the kwargs would pass in an object named that whose type would basically be a dataclass containing meta,body,spec, etc in one object?

E.g.

pod_type = ('', 'v1', 'pods)
@kopf.on.child_changed('zalando.org', 'v1', 'KopfExample', children={"pods": pod_type})
async def pods_changed(spec, pods, **kwargs):
    if is_correct(pods.name, pods.namespace, pods.spec):
        return
    # Spec is the KopfExample!
    update_it(spec, pods.name, pods.namespace)      

Worth further consideration:

• how to form the link in the first place (e.g. in on_create),
• whether a new event type is really required
• how to handle multiple children
• probably a bunch of complexities I've glossed over. :)

[elemental-lf]

First of all: Thank you for starting kopf! I actually started to roll my own solution based on https://github.com/side8/k8s-operator when I thankfully came across this project which already is so much better than anything I would have come up with. I just wanted to note that my use case would really profit from this idea:

The idea, in English, is to trigger a handler and to do things on MyDeployment when something happens on its children Deployment (or e.g. Pod).

But I also have some external state which I'd like to represent in Kubernetes and for that @kopf.on.resume and the planned @kopf.on.timer could be used to keep everything in sync without resorting to an extra thread/task. In this context an admission controller that could prevent certain object changes by a user would also be useful I think. I've read in another issue that this might be in scope for this project. I'm thinking about emulating such behaviour by simply reverting certain changes a user does.

[nolar]

@elemental-lf Thanks.

The current roadmap is:

1. Existing PRs for filtering the events via decorator keywords.
2. Stability (specifically, proper termination). Includes the full async/await inside (aiohttp).
3. Cross-object communication and further handlers/decorators extensions.

The number 1 is waiting in a PR. The number is 2 is done, being prepared for PRs. So, the objects relations are the next step in Kopf's roadmap and new feature additions.

The admission controllers are not currently in the near-future plans. However, as part of item 2, there comes a liveness-checker with HTTP port and a minimal web server (no data served, just liveliness). That can simplify adding the verifying/mutating admission controller at the next step after the current scope is done. No promises yet, but you can expect it coming.

[embano1]

But I also have some external state which I'd like to represent in Kubernetes and for that @kopf.on.resume and the planned @kopf.on.timer could be used to keep everything in sync without resorting to an extra thread/task.

Exactly, that would be very neat! Something I just discovered with kopf when trying to implement a ResyncPeriod I use in the Go SDKs.

[nolar]

Daemons and timers are added in kopf>=0.27rc1.

Release notes:

• https://github.com/zalando-incubator/kopf/releases/tag/0.27rc1

Docs:

• https://kopf.readthedocs.io/en/latest/daemons/
• https://kopf.readthedocs.io/en/latest/timers/

This is not yet cross-resource relation handlers for proper event-/state-driven reconciliation, but it can help with reconciliation by regular polling as discussed above:

• https://kopf.readthedocs.io/en/latest/reconciliation/

Beware: the changes are quite massive, and the daemons/timers are a new feature, they are not tested thoroughly yet, can behave weird. Either test carefully in an isolated cluster/namespace, or wait for some time until the feature is tested by others (e.g. by us).