Kuma support for multi-zone service-mesh deployments

[mhumeSF]

trafficstars

Describe the feature

Kuma support for multi-zone (multi-cluster) service-mesh deployments

What problem are you trying to solve? Currently when using flagger with kuma set as the mesh-provider, updates to the service mesh are sent to the kuma control plane on the same cluster where flagger is deployed. When kuma is deployed in a multi-zone deployment mode, updates have to be sent to the global control plane.

Currently flagger reports errors when sending update instructions:

{"level":"info","ts":"2022-04-13T22:33:15.403Z","caller":"controller/events.go:45","msg":"TrafficRoute podinfo create error: admission webhook \"validator.kuma-admission.kuma.io\" denied the request: Operation not allowed. Kuma resources like TrafficRoute can be updated or deleted only from the GLOBAL control plane and not from a ZONE control plane.","canary":"podinfo.test"}

Proposed solution

We need a way to tell flagger the address of the kuma global control plane

[lahabana]

Have you tried using --kubeconfig-host with a kubconfig for the global cluster? Seems like this should work. In any case we should update the docs to explain this

[mhumeSF]

I'll give this a try and report back

[michaelbeaumont]

It looks like setting the arg --kubeconfig-service-mesh to a path containing a kubeconfig file for the global cluster works. Flagger running on the zone cluster then creates policies via the global CP.

{"level":"info","ts":"2022-05-03T09:53:59.509Z","caller":"router/kuma.go:111","msg":"TrafficRoute podinfo created","canary":"podinfo.test"}

Given a secret global-kubeconfig with a key kubeconfig containing the kubeconfig, with Helm you'll want to set Values.istio.kubeconfig.secretName and Values.istio.kubeconfig.key to global-kubeconfig and kubeconfig.

[lahabana]

The Values.istio.kubeconfig.secretName istio bit is weird here :) Might be worth committing a change to rename this config in the helm chart no?

[stefanprodan]

@michaelbeaumont have you tested the kubeconfig flag? Based on this reporting seems broken https://github.com/fluxcd/flagger/issues/694

[michaelbeaumont]

@stefanprodan I verified that Kuma policies are at least able to be created. It's not clear to me yet in what way that Istio problem would manifest using Kuma but I'll take a deeper look.

[stefanprodan]

Because we add owner refs to the generated objects, and the owner object (the canary) is not present on that cluster, only the TrafficRoutes are.

[michaelbeaumont]

Kuma policies are cluster-scoped so I don't think a TrafficRoute can get an owner reference to a Canary which is namespaced. The TrafficRoute is not recreated in a loop like in the istio issue.

[stefanprodan]

@michaelbeaumont ah good point, thanks for the explanation.