k0s
k0s copied to clipboard
k0sproject
Allow replace kube-proxy by kube-router Service Proxy
Is your feature request related to a problem? Please describe.
In Lightweight environments, kube-proxy may be replaced by kube-router service proxy (--run-service-proxy). This mode uses IPVS (And this way, it's directly compatible with MetalLB - even more if flag --advertise-loadbalancer-ip got added to ClusterConfig) and uses just one service to provide networking.
Describe the solution you would like
No response
Describe alternatives you've considered
No response
Additional context
No response
There's a related discussion on the forums. The gist of it: You can try to configure k0s in this way, by disabling kube-proxy (--disable-components=kube-proxy) and deploying kube-router as a custom CNI.
Note that in the upcoming k0s 1.29 release, there will be the ability to pass extra arguments to the k0s-managed kube-router deployment. You may be able to get by without deploying kube-router yourself by adding --run-service-proxy to the kube-router extraArgs section of the k0s config. See #3902 for details.
There's a related discussion on the forums. The gist of it: You can try to configure k0s in this way, by disabling kube-proxy (
--disable-components=kube-proxy) and deploying kube-router as a custom CNI.Note that in the upcoming k0s 1.29 release, there will be the ability to pass extra arguments to the k0s-managed kube-router deployment. You may be able to get by without deploying kube-router yourself by adding
--run-service-proxyto the kube-router extraArgs section of the k0s config. See #3902 for details.
Hello!
I tried a variant of this, editing daemonset to add --run-service-proxy by my own and disabling kubeproxy on spec.network. But for some reason, this did not worked as expected because kube-router did not configured ipvs as expected. Also, I see k0s uses 1.x kube-proxy version instead 2.x and I dont know if this changes something.
About #3902 - this commit is not expected to be released on 1.28 series?
Also, there is any issue of using kube-router for service-proxy as default if user disables kube-proxy? There is some advantage to keep kube-proxy instead using by default kube-router service-proxy suppport?
Also, I see k0s uses 1.x kube-proxy version instead 2.x and I dont know if this changes something.
I guess you mean kube-router, not kube-proxy? The update is being worked on (#3814).
About #3902 - this commit is not expected to be released on 1.28 series?
This is not planned. The 1.29 release will not take too long anymore.
Also, there is any issue of using kube-router for service-proxy as default if user disables kube-proxy? There is some advantage to keep kube-proxy instead using by default kube-router service-proxy suppport?
I'll refer to @juanluisvaladas's answer: http://forums.k8slens.dev/t/how-to-enable-kube-routers-load-balance-feature-under-k0s/800/4
1.29 release will have new feature which allows you to configure any args for kube-router. With that you can the enable proxy in kube-router and disable k0s setting up kube-proxy.
To keep the scope on track, I suggest we close this issue since everything is tracked here.
But a question that @juanluisvaladas awnser got me: If kube-proxy implements LoadBalancer, why this is not mentioned in https://docs.k0sproject.io/v1.28.5+k0s.0/examples/nginx-ingress/?h=loadbalance#install-nginx-using-loadbalancer as example?
In background (But will not ask this here now) i'm investigating a misbehave of any IPVS implementation with MetalLB (SYN goes, SYN+ACK not and I see returning packets does not have LB IP as source, but POD IP) and i'm thinking now if use kube-proxy as LB class may fix this. In this case, kube-proxy LB capability does not deserve any additional documentation?
Hi @leleobhz,
But a question that @juanluisvaladas awnser got me: If kube-proxy implements LoadBalancer, why this is not mentioned in https://docs.k0sproject.io/v1.28.5+k0s.0/examples/nginx-ingress/?h=loadbalance#install-nginx-using-loadbalancer as example?
Neither kube-proxy or kube-router implement LoadBalancer type, I did say that, but it must have been a lapse because LoadBalancer is intended for external load balancers and kube-proxy was never supposed to implement it. It was 6 months ago so I'm not sure what was the missing feature I was mentioning but I remember there were a couple...
Anyway, what I said is we're willing to study feature requests as long as they are justified, the previous request didn't say why we should implement but this but you are providing some reasons to do so. So it's definitely worth considering this again.
Do you think that, given that you can disable kube-proxy and starting 1.29 you'll be able to add the flag this is still needed? Or having that suffices?
As for the kube-proxy version, @twz123 mentioned you may mean kube-router, but if you are not please clarify exactly what are you referring to, I'm aware of kpng and KEP-2104 but I don't think you're referring to that...
Hi @leleobhz,
Hi @juanluisvaladas
Anyway, what I said is we're willing to study feature requests as long as they are justified, the previous request didn't say why we should implement but this but you are providing some reasons to do so. So it's definitely worth considering this again.
I'm thankful by your consideration about this. I'm challenging myself run k0s on Rpi3 cluster because 1) k0s is the most upstream-compliant mini-distro of kubernetes I've found 2) I want to study how Kubernetes can be improved on low memory scenarios (And I got good result with NodeSwap + zram + more disk swap) 3) Learn more about Kubernetes internal. Also, Kubernetes on low requisite scenarios is a important study.
Do you think that, given that you can disable kube-proxy and starting 1.29 you'll be able to add the flag this is still needed? Or having that suffices?
I'll attempt this after 1.29 release. I'm stuck on tricky network issue here (No matter what, anything using LoadBalancer class starts 3 way handshake but after 3rd ack awnser, communication stops and I receive a tons of retransmissions until timeout) but if I can reach this point using only Kuberouter, I think it's possible consider it works. Also, I'll try change distro/kernel to check for some heavy restriction on environment (DietPI is a lot compact but may be missing something).
As for the kube-proxy version, @twz123 mentioned you may mean kube-router, but if you are not please clarify exactly what are you referring to, I'm aware of kpng and https://github.com/kubernetes/enhancements/issues/2104 but I don't think you're referring to that...
Indeed I mean kube-router version: https://github.com/k0sproject/k0s/blob/2a8c296c9121752c7f30a047eb6ec8597762819b/pkg/constant/constant_shared.go#L91
I think if Kube-router got updated to 2.x line, maybe possible to use only kube-router + metallb with kube-router provinding pod networking + IPVS proxy + BGP and metallb providing LB Class (kube-router manages BGP and with the bgp-announce-lb-ip as true it manages lb address announcement too). And on implementations that does not rely on external BGP, kuberouter can provide pod networking + IPVS proxy and metallb with LB Class plus ARP listener.
For runnning with only standalone kuberouter, kuberouter daemonset needs a proper configured kubeconfig Arg ,
this is not implemented in
https://github.com/k0sproject/k0s/blob/f29519bc3c9a8c09236f8384d5181eaa619d5764/pkg/component/controller/kuberouter.go#L183
like it it is for kubeproxy ( cleaner > server: {{ .ControlPlaneEndpoint }} )
https://github.com/k0sproject/k0s/blob/33b4f8ad4952b137236e854c494a80634425d8c6/pkg/component/controller/kubeproxy.go#L265
so only got it working with custom network provider option,and with the hardcoded internal api server IP and port 6443 on the manifest kubeconfig, ( hacky at best )
still not convinced of the benefits of disabling kube-proxy.
still not convinced of the benefits of disabling kube-proxy.
Hello @pedro-n-rocha
Main reason is system resource waist. Calico replaces kube-proxy in eBPF mode and kube-router also does support replace kube-proxy. Not telling here kube-proxy is a bad implementation, but maybe redundant in some scenarios. It's at least a good look into flexibility since k0s can be customized/extended with helm and Calico - as example - can be deployed and configured entirelly from helm charts.
@leleobhz Now that k0s supports adding the needed arg(s) to kube-router and kube-proxy can be disabled, I don't think there's anything more for k0s to do, right? If so, pls close the issue
Hi @jnummelin
I've tried this in a older version of k0s. I'll try to replicate this on 1.31 and I'll feedback here.
Can someone show how to pass arguments on kuberouter please
A simple case:
kuberouter
extraArgs:
- '--run-service-proxy=true'
but it complains because it expects it as map[string]string
+++++++++++++++++++++++++ I found it
extraArgs:
run-service-proxy: "true"
Sorry for the noise
It is not sufficient to pass the extraArgs for KubeRouter, as it also requires a kubeconfig.conf file to communicate with the API server.
Currently, this functionality is not implemented in the k0s kubeRouterTemplate variable in kuberouter.go, unlike the proxyTemplate variable in kubeproxy.go, which handles this appropriately.
As a temporary solution, I’ve disabled the default KubeRouter handling in k0s by setting:
k0s controller -c /etc/k0s/config.yaml --enable-worker --single --disable-components=autopilot,endpoint-reconciler,helm,konnectivity-server,metrics-server,windows-node,kube-proxy
And configured the network provider as custom in the k0s_config environment variable:
Then, I’ve placed a fully configured kube-router.yaml manifest under:
/var/lib/k0s/manifests/kuberouter/
This workaround ensures KubeRouter operates correctly, but it would be better to implement proper support for generating and managing the kubeconfig.conf file within the kubeRouterTemplate logic.
This is a straightforward fix that would bring parity between kubeproxy.go and kuberouter.go and improve the developer and user experience.
Currently, I'm having a k0s setup with this k0sctl:
k0s:
config:
apiVersion: k0s.k0sproject.io/v1beta1
kind: Cluster
metadata:
name: k0s
spec:
api:
port: 8042
k0sApiPort: 8043
konnectivity:
adminPort: 8048
agentPort: 8049
network:
provider: kuberouter
kuberouter:
metricsPort: 8047
autoMTU: true
hairpin: Enabled
kubeProxy:
disabled: false
mode: nftables
podCIDR: 10.64.0.0/16
serviceCIDR: 10.48.0.0/12
installConfig:
users:
etcdUser: etcd
kineUser: kube-apiserver
konnectivityUser: konnectivity-server
kubeAPIserverUser: kube-apiserver
kubeSchedulerUser: kube-scheduler
podSecurityPolicy:
defaultPolicy: 00-k0s-privileged
storage:
type: etcd
network setup with kube-proxy + kube-router works well:
k8s ~ # kubectl get all -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system pod/coredns-6946cc8786-ssbqx 1/1 Running 0 7m59s
kube-system pod/konnectivity-agent-zpgtl 1/1 Running 0 7m55s
kube-system pod/kube-proxy-p4gjc 1/1 Running 0 7m58s
kube-system pod/kube-router-m5mfz 1/1 Running 0 7m58s
kube-system pod/metrics-server-7db8586f5-r6zdf 1/1 Running 0 7m55s
...
k8s ~ # ss -ltunp | grep -E 'kube|etc|k0s'
tcp LISTEN 0 4096 127.0.0.1:50051 0.0.0.0:* users:(("kube-router",pid=3046,fd=10))
tcp LISTEN 0 4096 192.168.3.3:179 0.0.0.0:* users:(("kube-router",pid=3046,fd=9))
tcp LISTEN 0 4096 192.168.3.3:2380 0.0.0.0:* users:(("etcd",pid=1901,fd=6))
tcp LISTEN 0 4096 192.168.3.3:50051 0.0.0.0:* users:(("kube-router",pid=3046,fd=8))
tcp LISTEN 0 4096 127.0.0.1:2379 0.0.0.0:* users:(("etcd",pid=1901,fd=7))
tcp LISTEN 0 4096 127.0.0.1:10248 0.0.0.0:* users:(("kubelet",pid=2358,fd=14))
tcp LISTEN 0 4096 127.0.0.1:10257 0.0.0.0:* users:(("kube-controller",pid=2189,fd=3))
tcp LISTEN 0 4096 127.0.0.1:10259 0.0.0.0:* users:(("kube-scheduler",pid=2188,fd=3))
tcp LISTEN 0 4096 *:20244 *:* users:(("kube-router",pid=3046,fd=3))
tcp LISTEN 0 4096 *:10249 *:* users:(("kube-proxy",pid=2583,fd=8))
tcp LISTEN 0 4096 *:10250 *:* users:(("kubelet",pid=2358,fd=20))
tcp LISTEN 0 4096 *:10256 *:* users:(("kube-proxy",pid=2583,fd=9))
tcp LISTEN 0 4096 *:8047 *:* users:(("kube-router",pid=3046,fd=6))
tcp LISTEN 0 4096 *:8042 *:* users:(("kube-apiserver",pid=1927,fd=3))
tcp LISTEN 0 4096 *:8043 *:* users:(("k0s",pid=2059,fd=3))
I'd like to remove kube-proxy completely with:
config:
apiVersion: k0s.k0sproject.io/v1beta1
kind: Cluster
metadata:
name: k0s
spec:
api:
port: 8042
k0sApiPort: 8043
konnectivity:
adminPort: 8048
agentPort: 8049
network:
provider: kuberouter
kuberouter:
metricsPort: 8047
autoMTU: true
hairpin: Enabled
extraArgs:
run-service-proxy: "true"
run-firewall: "true"
run-router: "false"
kubeProxy:
disabled: true
mode: nftables
podCIDR: 10.64.0.0/16
serviceCIDR: 10.48.0.0/12
installConfig:
users:
etcdUser: etcd
kineUser: kube-apiserver
konnectivityUser: konnectivity-server
kubeAPIserverUser: kube-apiserver
kubeSchedulerUser: kube-scheduler
podSecurityPolicy:
defaultPolicy: 00-k0s-privileged
storage:
type: etcd
However this never works, kube-router will crash soon:
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system pod/coredns-6946cc8786-mtqk6 0/1 ContainerCreating 0 2m26s
kube-system pod/konnectivity-agent-2pdn9 0/1 ContainerCreating 0 2m25s
kube-system pod/kube-router-w6ss2 1/1 Running 1 (39s ago) 2m25s
kube-system pod/metrics-server-7db8586f5-dxp62 0/1 ContainerCreating 0 2m22s
anyone has a reference guide for such a setup, appreciate a lot
Hello @rtgiskard
I did it sucessfully this time with following parts:
- Network config:
network:
clusterDomain: <REDACTED>
podCIDR: "10.244.0.0/16"
serviceCIDR: "10.96.0.0/12"
provider: custom
dualStack:
enabled: true
IPv6podCIDR: "fd00::/108"
IPv6serviceCIDR: "fd01::/108"
kubeProxy:
disabled: true
- Helm for calico with eBPF:
config:
kind: ClusterConfig
metadata:
creationTimestamp: null
name: <REDACTED>
spec:
extensions:
helm:
concurrencyLevel: 1
repositories:
- name: tigera-operator
url: https://docs.tigera.io/calico/charts
charts:
- name: tigera-operator
chartname: tigera-operator/tigera-operator
order: 1
namespace: tigera-operator
version: v3.30.3
values: |
goldmane:
enabled: false
whisker:
enabled: false
kubernetesServiceEndpoint:
host: "<KUBEAPI_IP>"
port: "6443"
kubeletVolumePluginPath: /var/lib/k0s/kubelet
defaultFelixConfiguration:
enabled: true
bpfExternalServiceMode: DSR
prometheusGoMetricsEnabled: true
prometheusMetricsEnabled: true
prometheusProcessMetricsEnabled: true
installation:
enabled: true
cni:
type: Calico
calicoNetwork:
linuxDataplane: BPF
bgp: Enabled
ipPools:
# ---- podCIDRv4 ---- #
- cidr: 10.244.0.0/16
name: podcidr-v4
encapsulation: VXLANCrossSubnet
natOutgoing: Enabled
# ---- podCIDRv6 ---- #
- cidr: fd00::/108
name: podcidr-v6
encapsulation: VXLANCrossSubnet
natOutgoing: Enabled
# ---- PureLBv4 ---- #
- cidr: 192.168.50.0/24
name: purelb-v4
disableNewAllocations: true
# ---- PureLBv6 ---- #
- cidr: fd53:9ef0:8683:50::/120
name: purelb-v6
disableNewAllocations: true
# ---- EOF ---- #
nodeAddressAutodetectionV4:
interface: "br0"
nodeAddressAutodetectionV6:
cidrs:
- fc00:d33d:b112:50::0/124
calicoNodeDaemonSet:
spec:
template:
spec:
tolerations:
- effect: NoSchedule
operator: Exists
- effect: NoExecute
operator: Exists
csiNodeDriverDaemonSet:
spec:
template:
spec:
tolerations:
- effect: NoSchedule
operator: Exists
- effect: NoExecute
operator: Exists
calicoKubeControllersDeployment:
spec:
template:
spec:
tolerations:
- effect: NoSchedule
operator: Exists
- effect: NoExecute
operator: Exists
typhaDeployment:
spec:
template:
spec:
tolerations:
- effect: NoSchedule
operator: Exists
- effect: NoExecute
operator: Exists
tolerations:
- effect: NoSchedule
operator: Exists
- effect: NoExecute
operator: Exists
Please note you WILL NEED TO CHANGE calico behavior about interface detection and networks (I use purelb as LoadBalance provider). Basically this example deploys tigera-operator to deploy Calico as operator with my own config and eBPF enabled (Removing kube-proxy needs) and set k0s to not use kube-proxy.
@leleobhz Thanks for sharing! Calico seems a bit too complex for my use case. I’d prefer kube-router as it’s an all-in-one solution, lightweight, and doesn’t require additional setup
Hello @rtgiskard !
Replace kube-proxy will never happen w/o additional setup unless k0s upstreams this request (And I particularly dont think it will happen and I dont mind why to do since k0s configuration is flexible enough to handle this).
That said, Calico on my example setup is complex as it is on these manifests. I use BGP - you may dont need it, I use eBPF - you also may don't need it because eBPF is not the unique way Calico handles proxy and I use PureLB (A simpler implementation than MetalLB right because Calico handles my LB entries and BGP Announces). So in a scenario you just replace kube-proxy+kube-router by Calico the config will get simpler than mine.
The all-in-one solution by k0s is today kube-proxy+kube-router. Everything else is up to you.
EDIT: typo
Just for the reference, there is no need to provide a new kubeconfig, you can just grant additional permissions to the service account: https://github.com/k0sproject/k0s/issues/6340#issuecomment-3244951634
Hello @rtgiskard !
Replace kube-proxy will never happen w/o additional setup unless k0s upstreams this request (And I particularly dont think it will happen and I dont mind why to do since k0s configuration is flexible enough to handle this).
That said, Calico on my example setup is complex as it is on these manifests. I use BGP - you may dont need it, I use eBPF - you also may don't need it because eBPF is not the unique way Calico handles proxy and I use PureLB (A simpler implementation than MetalLB right because Calico handles my LB entries and BGP Announces). So in a scenario you just replace kube-proxy+kube-router by Calico the config will get simpler than mine.
The all-in-one solution by k0s is today kube-proxy+kube-router. Everything else is up to you.
EDIT: typo
Finally, I'm moving to cilium, great for everything! no kube-router, no kube-proxy!