metacontroller

metacontroller copied to clipboard

One of the things I've been bumping up against when trying to design a controller is incorporating the state of related objects.

For context - we have many environments, and are trying to have a strict separation between the application bundle of manifests, which is common to all (and can be released/rolled out etc), and the site specific manifests, which are mostly ConfigMaps / Secrets.

One of the things I'm thinking about how to do is to write a controller that can use ConfigMaps. A concrete example would be designing a new object incorporating something akin to an EnvVar array. The desired outcome would be that a CompositeController can indicate in some fashion that it's interested in the state of some related objects, and be called whenever those change as well.

A possible implementation might be that the sync hook can return an additional field of "related" (better name?) that indicates objects it's interested in. This might be explicit objects, or label selectors / types. Then metacontroller can call the sync hook again, with the additional information included, to allow the hook to make the right decision.

An interesting consideration is that if metacontroller restarts, and thus forgets about which additional objects are required, the sync hook may be unable to return an appropriate response (i.e. it may now now know if some of the objects it's being told about it owns are needed or not). As it's recommended that the sync hook doesn't simply copy in existing objects verbatim, but only specifies the fields it cares about, this may be problematic. A possible solution is that the hook returns status code 510, which seems to match up. In such a case metacontroller should ignore the returned status and children fields and retry with the requested information.

Alternatively it might be better to focus on writing MutatingAdmissionWebhooks to add additional fields into certain objects based on our environment. I'm not sure which would be a more sensible approach.

(Certainly we're going to have to go with MutatingAdmissionWebhooks for say, adding a site: blah label to all of our Prometheus objects, for instance. The question is more whether we can make our own resources easier to use in that regard)

I am also interested in this behavior. My example is a CompositeController that generates a child Pod with values from an existing ConfigMap that it doesn't own. I would want to use the data from the ConfigMap as part of the pod definition and I want to update the pod if the ConfigMap contents change.

I've been working around this today by calling the k8s API from my webhook to fetch and sync the related objects. This doesn't scale though as one of the advantages of metacontroller is that it brokers requests to the apiserver...

One idea would be to have a related field in the CompositeController spec that is a list of object selectors. The sync request could then include those selected objects so they can be used in the webhook. This might not be the best solution as you may not know what the related object selectors are at the time you define the CompositeController.

Another idea that I'm playing with is using a DecoratorController as a child of a CompositeController. The decorator controller would select the related objects used by the CompositeController resource and communicate them to the webhook through a shared cache like Redis. This would at least prevent additional calls to the apiserver from the webhook, but it adds some complexity.

I was thinking about this a little more and it might be possible to enhance metacontroller to provide the related objects in the CompositeController webhook sync request using a sidecar CRD, similar to how the BackendConfig resource for the Google Cloud Load Balancer works.

For example, you could claim a child object in your controller that instructs the metacontroller on the related objects you are interested in:

apiVersion: metacontroller.k8s.io/v1alpha1
kind: RelatedObjects
metadata:
  name: my-objects
spec:
  resources:
  - apiVersion: v1
    resource: pods
    annotationSelector:
      matchExpressions:
      - {key: pod-name-label, operator: Exists}

The owner reference would then be set to the parent object by metacontroller because it's a child resource. Metacontroller could then configure the informers and caches to observe and include the complete objects for the selected resources in the parent webhook sync request.

@mikebryant wrote:

The desired outcome would be that a CompositeController can indicate in some fashion that it's interested in the state of some related objects, and be called whenever those change as well.

This sounds similar to the "Friend Resources" alternative described in a proposal I wrote about another pattern for handling "related resources":

https://metacontroller.app/design/map-controller/#friend-resources

I ended up not exploring friend resources further, but if it fits your use case better we can always reconsider modifications to address the downsides listed in the proposal. Please take a look at the other thoughts in that proposal and let me know if anything resonates with you.

@danisla wrote:

My example is a CompositeController that generates a child Pod with values from an existing ConfigMap that it doesn't own. I would want to use the data from the ConfigMap as part of the pod definition and I want to update the pod if the ConfigMap contents change.

If there's a 1:1 correspondence between ConfigMap -> Pod, then I think you can do this with the proposed MapController pattern. Can you take a look at that and let me know if you agree?

For example, you could claim a child object in your controller that instructs the metacontroller on the related objects you are interested in [...] Metacontroller could then configure the informers and caches to observe and include the complete objects for the selected resources in the parent webhook sync request.

This would mean that the set of resources you watch could change from one sync to the next. Is that really a requirement for the use case? We could certainly support that, but it would be a significant added layer of complexity. Is there a reason you couldn't know what resources you need at the time of writing the CompositeController spec?

If there's a 1:1 correspondence between ConfigMap -> Pod, then I think you can do this with the proposed MapController pattern. Can you take a look at that and let me know if you agree?

In my case it could be a many-to-one relationship. My composite controller may need to know about the content of multiple ConfigMaps and Secrets before it can generate the child Pod. It sounds like MapController would operate on a single input non-owned resource.

Is there a reason you couldn't know what resources you need at the time of writing the CompositeController spec?

I could define some convention in my CompositeController spec that would dictate a set of annotations or labels for related resources but then every instance of the parent would receive every matching object. In my case, users would be providing say a list of specific ConfigMap selectors for that particular instance of a parent, which is why I don't necessarily know exactly what related objects I want at the time I create the CompositeController spec.

I've been running into this more when trying to design an operator for PagerDuty.

I want to have a resource PagerDutyAccount, which has a secretRef to where the account key is held.

And then several other types that all have a pagerDutyAccountRef, which the operator can use to contact the API. At the moment this is problematic, as the hook doesn't send the related object, or notify on changes etc.

@mikebryant wrote:

A possible implementation might be that the sync hook can return an additional field of "related" (better name?) that indicates objects it's interested in. This might be explicit objects, or label selectors / types. Then metacontroller can call the sync hook again, with the additional information included, to allow the hook to make the right decision.

After thinking about this some more, I'm coming around to this idea.

An interesting consideration is that if metacontroller restarts, and thus forgets about which additional objects are required, the sync hook may be unable to return an appropriate response (i.e. it may now now know if some of the objects it's being told about it owns are needed or not).

I'm thinking this can be avoided if we split the new question ("which related objects do you need?") into its own hook, which we would call prior to sync. That would mean an extra round trip, but the actual number of such extra calls could be much lower than the number of sync calls if we restrict you to only deciding which objects you need based on the parent spec. That is, you can't change your answer based on the states of children, nor on the states of objects you previously specified were related.

In concrete terms, the related hook will send you only the parent spec (no children, no related objects), and it will only get called again if the parent spec changes. In most cases, I'd expect the majority of sync calls to be triggered by changes in the child objects; the number of syncs triggered by changes to the parent is usually much lower.

This keeps the sync request/response contract clean: we'll send you everything you need, and you should always return a real desired state (as opposed to a special error code that means some parts of the response are valid but not others).

Having a separate hook would be fine. All the use-cases I'm imagining depend only on the .spec

FYI. I have tried to implement this as GenericController in a forked version of MetaController. This is a completely new meta controller on the lines of existing Composite & Decorator controllers.

refer - https://github.com/AmitKumarDas/metac/ api - https://github.com/AmitKumarDas/metac/blob/master/apis/metacontroller/v1alpha1/types_generic.go