Multiple peer DB-less controllers attempt Programmed status updates

[rainest]

trafficstars

Overview

Broken out of a comment on another issue.

https://github.com/Kong/kubernetes-ingress-controller/pull/4412 introduced statuses for KongConsumer. When deploying a controller instance per Kong instance in DB-less mode (the "legacy" configuration from before gateway discovery), each instance of the controller will attempt to set this status.

In legacy configuration, controllers do not use leader election. Because each controller instance needs to update its partner Kong instance, each controller must run every reconciler. The reconcilers apply status updates.

While determining the exact sequence of events would require tedious log review, basic observation indicates that it's roughly:

1. Controllers will attempt to initially set programmed to false. They will race until one sets it, after which the others will fail to set it and log an error.
2. Some controller will pass its update tick and set programmed to true. This triggers a reconcile. Any controller that has not yet passed their update tick will attempt to set it to false, triggering more reconciles.
3. Eventually all controllers will pass their update tick and all will want to set it to true, so the value will stop flapping and the updates will cease.

Replication:

https://raw.githubusercontent.com/Kong/kubernetes-ingress-controller/v2.11.1/deploy/single/all-in-one-dbless-legacy.yaml
kubectl scale deploy -n kong ingress-kong --replicas=3

In separate terminals:

kubectl logs -f -n kong -l app=ingress-kong -c ingress-controller | grep kotenok

echo "apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
  name: kotenok
  annotations:
    kubernetes.io/ingress.class: kong
username: kotenok
" | kubectl apply -f -

You should see multiple logs indicating update failures:

time="2023-08-31T22:19:10Z" level=error msg="Reconciler error" error="Operation cannot be fulfilled on kongconsumers.configuration.konghq.com \"kotenok\": the object has been modified; please apply your changes to the latest version and try again" logger=controllers.KongConsumer reconcileID="\"9920ed48-700e-4e14-9385-44c2b8e27384\""
time="2023-08-31T22:19:10Z" level=error msg="Reconciler error" error="Operation cannot be fulfilled on kongconsumers.configuration.konghq.com \"kotenok\": the object has been modified; please apply your changes to the latest version and try again" logger=controllers.KongConsumer reconcileID="\"d65f4772-7fd4-4b33-baab-c161056e5766\""
time="2023-08-31T22:19:10Z" level=error msg="Reconciler error" error="Operation cannot be fulfilled on kongconsumers.configuration.konghq.com \"kotenok\": the object has been modified; please apply your changes to the latest version and try again" logger=controllers.KongConsumer reconcileID="\"c17119a7-42ee-4c9c-8ebf-acc10e2c94ab\""

You can also observe the sequence of changes by creating the KongConsumer with a different class, running

watch -n 0.1 -d kubectl get kongconsumer kotenok -oyaml

and then editing the class to kong

[rainest]

This issue affects any resource that uses the current Programmed condition code (I did not exhaustively review to confirm, but AFAIK we copy/paste the same code for all resources that use it). GWAPI resources that use it (basically all of them) also exhibit this issue. We likely did not see prior reports because GWAPI usage is less common.

The Programmed condition is always set. Resources not loaded into the proxy get a false condition; resources that are get a true condition.

By contrast, Ingress status IPs are only set if the resource is configured in the gateway. If not configured, the controller does not modify the resource and just queues another reconciliation until it is. Controllers that update after the first will not attempt to remove the status, and will see that the status is already set to the desired value when they are ready to update.

[rainest]

Addressing this issue is tricky. I don't think there's any way around needing to run all reconcilers when using a controller per instance.

We can configure reconcilers to run without having leadership in leader elect mode, and could split them into two reconcilers for each kind, one which updates the store (always on, to create proxy configuration) and a second that updates the status (requires leadership). However, we also need to account for the existing modes that use leadership (DB-backed and DB-less with discovery), as we do not want those to update configuration.

I think we may already have some code in place that makes the update loop a LeaderElectionRunnable, but we'd need to confirm exactly what does and does not run when not a leader.

A simpler option is to disable Programmed updates in general for leaderless modes. Only having one of the peer instances update status is already a bit of a lie since there's no guarantee the config is actually universally available (in practice, it almost always is after the update tick has elapsed since the status update). Doing this would mean legacy configuration is not in compliance with the GWAPI spec, which requires those statuses.

@mflendrich briefly mentioned suppressing errors as an option. We don't currently have any interception logic (we just pass the Update() error back to controller-runtime's reconciler runner, which is what logs the error), but probably could add some. However, doing so would still thrash the API server with unnecessary requests. These aren't as bad as the classless case, where controllers will fight over the status indefinitely, but they're still best avoided. Reconciles run quickly enough that there's a surprisingly large number of them during the short window before instances converge, and this worsens as instance count increases.

[AlexZhenWang]

Hi I am getting the same issue, does anyone know any workaround to fix this issue?

[pmalek]

@AlexZhenWang What version are you using? This should be fixed in 2.11.1 as per the changelog https://github.com/Kong/kubernetes-ingress-controller/blob/main/CHANGELOG.md#2111

[danopia]

I am using 2.12 and see reconcile errors on a KongConsumer resource. It looks like 2.11.1 only reverted the addition for KongPlugin and KongClusterPlugin as per the linked changelog

[sharavara]

I have the same error with the KongConsumer resource. My version is 2.11

[davidmontoyago]

My team is also observing this behavior with KongConsumer resources on 2.12.7.