Enable kube-proxy replacement if no kube-proxy is detected in a cluster

[brb]

trafficstars

When installing with cilium install, the cilium-cli could detect whether K8s' kube-proxy is present in a cluster. If not, then it could install Cilium with --set kubeProxyReplacement=strict.

[ksankeerth]

Hi @brb, I'm new to cilium, Can I work on this feature?

[brb]

@ksankeerth Sure, thanks!

[ksankeerth]

Hi @brb,

Small help on this issue. To detect kube-proxy has been installed or not, I'm thinking to use ListDaemonSet function and doing a string match(kube-proxy/regex check). Is there any better idea? Appreciate your feedback on this.

Thanks

[brb]

@ksankeerth I think listing DaemonSets in the kube-system ns, and then matching for kube-proxy should work on > 80% clusters. So yeah, I suggest to take this approach.

Another problem which needs to be solved as part of this change is to detect the kube-apiserver IP and port (https://docs.cilium.io/en/v1.12/gettingstarted/kubeproxy-free/#quick-start). Not sure what's the most reliable strategy, but one way is to find a control plane node for IP and 6443 as port (default).

[ksankeerth]

@ksankeerth I think listing DaemonSets in the kube-system ns, and then matching for kube-proxy should work on > 80% clusters. So yeah, I suggest to take this approach.

Another problem which needs to be solved as part of this change is to detect the kube-apiserver IP and port (https://docs.cilium.io/en/v1.12/gettingstarted/kubeproxy-free/#quick-start). Not sure what's the most reliable strategy, but one way is to find a control plane node for IP and 6443 as port (default).

Thanks @brb.

For the second problem, I used already available clients(similar client we used for sysdump), so I don't need to worry about the IP and port of the Master node. I'm doing some tests and will send a PR soon.

[ksankeerth]

Hi @brb, I was able to test and created a PR.1039. Please check if possible. Thanks

root@kcp:/home/ubuntu/cilium-cli# ./cilium install
ℹ️  Using Cilium version 1.12.0
🔮 Auto-detected cluster name: kubernetes
🔮 Auto-detected datapath mode: tunnel
🔮 Auto-detected kube-proxy has not been installed
ℹ️  Cilium will fully replace all functionalities of kube-proxy
ℹ️  helm template --namespace kube-system cilium cilium/cilium --version 1.12.0 --set cluster.id=0,cluster.name=kubernetes,encryption.nodeEncryption=false,kubeProxyReplacement=strict,operator.replicas=1,serviceAccounts.cilium.name=cilium,serviceAccounts.operator.name=cilium-operator,tunnel=vxlan

[brb]

@ksankeerth Hey, one user has reported that on a k3s installation there is no kube-proxy pod. Perhaps we should detect when cilium-cli is running against a k3s cluster, and then opt out from the KPR auto detection. Mind looking into it?

[ksankeerth]

@ksankeerth Hey, one user has reported that on a k3s installation there is no kube-proxy pod. Perhaps we should detect when cilium-cli is running against a k3s cluster, and then opt out from the KPR auto detection. Mind looking into it?

Thanks for letting me know. I'll look into this and get back to you.

[brb]

@ksankeerth One more thing which needs to be fixed is opting out from the KPR auto-detection if user sets the KPR flag (via --helm-set or --kube-proxy-replacement).

[ksankeerth]

@brb Hope you're doing well. I looked at these improvements. We can use annotations of the K3s server/node to detect k3s and opt-out from the KPR auto replacement. But regarding the 2nd requirement, I think we need to change the default value.

Currently, if we just run cilium install, k.params.KubeProxyReplacement will hold "disabled" as the default value. So We won't be able to find whether it was set by the user or the default value of cilium-cli. Would it be ok to introduce a new value "auto" for k.params.KubeProxyReplacement and keep it as the default value? Then we can detect the user-specified options.

[brb]

@ksankeerth :wave:

We can use annotations of the K3s server/node to detect k3s and opt-out from the KPR auto replacement.

:+1:

But regarding the 2nd requirement, I think we need to change the default value.

The problem here is that we need to detect whether the flag was explicitly set by a user or whether the default is used. To solve this, we could use https://pkg.go.dev/flag#Visit to figure out whether it was set. If it was not, then run the detection.

[xiaods]

@ksankeerth any progress on k3s server to detect k3s opt-out kube-proxy setting.

[ksankeerth]

@ksankeerth any progress on k3s server to detect k3s opt-out kube-proxy setting.

sorry for the delay. Just submitted a PR for this.