Use-case for reconciling external state

[mikebryant]

(As requested in the FAQ)

One idea I've been toying with is implementing a PagerDuty operator for kubernetes

Things like a PagerDutyService CRD. And then automatically hooking up the generated service endpoint with the AlertManager resource from prometheus-operator.

The main reason to not create the services manually is for us replicating the same set of services across multiple environments, and we're trying to make all the ways of declaring our infrastructure kubernetes manifest shaped, so deploying an app is just applying a bunch of manifests.

[enisoc]

Thanks for filing this! It's definitely a use case that I think is a good one for Metacontroller to tackle. People should not have to think in terms of reconciling control loops just to plug one thing into another thing. We should just ask them to provide a little glue.

The way I've been thinking about the general pattern is something like:

I want to create a Kubernetes-native, declarative API that's a wrapper of an existing, traditional, imperative API.

Once #60 is addressed, you'll be able to get most of the way there with CompositeController. You'll have the ability to reliably create, update, and delete things in the external system in response to Kubernetes resources. The biggest gap I see remaining after that will be pulling state changes that happen in the external system back into the Kubernetes API surface, so you can automate reactions to those changes.

You can approximate that by setting CompositeController to poll your sync hook, and having your hook in turn query the external system's API to check on the latest state there. However, I've been thinking this pattern might deserve its own Metacontroller API -- something like AdapterController. We could make this API better fit the use case of 2-way synchronization of state between Kubernetes objects and an external, imperative API.

As an example, perhaps AdapterController would have the ability to define inbound hooks, which you can wire up to something that knows when things change in the external state. These inbound hooks could look like plain webhooks, or perhaps bindings through something like Knative Eventing. Metacontroller would then know to sync your controller if state changes on either side -- in Kubernetes or in the external system.

As another example, AdapterController might have the ability to curate a continuously updated collection of top-level objects in a CRD (i.e. Metacontroller creates these objects; it's not just responding to objects the user creates) that represent external state. This would mean, for instance, you could expect to see PagerDuty services show up in kubectl get pagerdutyservices even if they weren't created via the PagerDutyService CRD (assuming you choose to establish such a connection).

Let me know if you have any other thoughts on how Metacontroller could help, or if something like AdapterController would fit your needs.

[mikebryant]

Thanks for the detailed response.

Yes, I can see how I can get to what I need with finalizers, thanks. (This will only automate part of our PagerDuty account, so there's always going to be unmanaged resources there created manually.) As long as I can clean up stuff I create I'm happy, I think.

Having something that can pull changes back is quite an interesting concept. I'm not actually sure inbound hooks would help in the case I'm thinking about - there's no event based api to watch for changes, so it would be polling anyway. I'm also struggling to think about how the reconciliation loop would work...

• if the controller treats the same objects in the api as both a desired state and also reflecting the state of the underlying resource we have issues. e.g. if someone creates a service manually, it shows up in the api. If they delete it, the controller would then just keep recreating it, which I think would be very unexpected.
• if the controller doesn't treat them as desired state then it doesn't help much - you'd need to have a separate object DesiredPagerDutyService, which then would have to create them directly in the pagerduty api, which then means they show up in Kubernetes API. It doesn't quite seem worth it.

I think I'm thinking of this more in terms of Pods in the kubelet, vs docker containers on disk. We make Pod objects, they end up making containers, but we never reflect in the kubernetes API any other containers that happen to exist (excepting mirror pods, but those also come from desired state, rather than containers managed outside of kubernetes)

...It's quite possible I just don't grok how you're anticipating that sort of reconciliation might work

tl;dr finalizers sound suitable for what I need, thanks for the pointer to #60

[lionelvillard]

I'm also interested in this pattern, except for the inbound hooks. DecoratorController almost fits the bill (thanks to #60), the only missing bit is updating the status of the target resource. Is this a reasonable extension to the DecoratorController?

[enisoc]

@lionelvillard The original reason for not letting DecoratorController update status is that it wouldn't work in most cases, with the assumption that DecoratorController would mostly be used to attach things to resources that already have their own controller (which would overwrite any status updates you made).

Are you thinking of using DecoratorController for a resource that doesn't already have another controller? If so, why not CompositeController?

Or are you thinking of using DecoratorCotnroller to update status of one of the rare resources that allows multiple controllers to collaborate on status (I think Pod recently added an alpha feature to allow this)?

[lionelvillard]

I'm thinking of using DecoratorController for resources without controllers, which is the case for custom resources. CompositeController could be used (not sure) but seems overkill and counter-intuitive.

[lionelvillard]

@enisoc I'm happy to contribute if you agree that's the right thing to do (adding a status field in the response)

[enisoc]

@lionelvillard OK, I'm convinced that it makes sense to let users try to update status in DecoratorController, since it seems people are finding Decorator useful even for cases when there's no existing controller for a resource. If you're able to work on that, I'd be happy to review it.