kube-router
kube-router copied to clipboard
cloudnativelabs
Consider supporting Service.Type=LoadBalancer
I'd envision it using some magic such as ECMP, but it doesn't much matter tbh how it works:
This project from google is a P.O.C. of doing something similar. This gives us using bare metal a way to have load balanced service ips using dedicated vips.
A less-elegant alternative is the keepalived-cloud-provider project. When doing a single AS per rack topology (like we are), this means you have to pin instances of this per rack, which is painful and not as resilient when standard BGP does a better job.
Its not hard to implement given Kube-router already does similar functionality for external IP's. Make sense to support it for completion.
I will take a shot at it. Adding support for --advertise-loadbalancer-ip is stright forward. How do we do IP address pool management for the services:
- with out taking any dependecy on etcd (ConfigMaps?)
- avoiding concurrency issue while allocating ip for the service any suggestions @bzub @SEJeff ?
Playing devil's advocate, how does the service ip range get handled? If I remember (without looking), it is managed by the controller manager. Does that just create Endpoint objects serially more or less?
One approach for this is something like Endpoint that is a CRD and a single (1 per cluster) controller to manage it. That doesn't omit the dependency on etcd, but kubernetes can't run without etcd, so why avoid it?
kubernetes can't run without etcd, so why avoid it?
I meant, Kube-Router not taking any dependency on etcd, like what calico does.
As a basic thing, you could just make kube-router program IPVS when it sees a Service with status.loadBalancer.ingress[*].ip . That field is what kube-proxy looks at to program correct routing for LoadBalancer services. The IPVS behavior should be exactly the same as externalIPs, so it should be a pretty trivial change in kube-router.
Once you've done that, kube-router should be compatible (at a basic level) with MetalLB, and also with all cloud providers like GCP, where a platform-specific component handles IP allocation.
And then you can separately decide if you want to implement your own IP allocators, or just say that other services like MetalLB provide that :)
@danderson thanks for your comments. Your input make perfect sense.
We will also revisit metallb integration to make it work with Kube-Router. There are users who wish to use kube-router just as service proxy so there is no reason why kube-router cannot work with metal LB (as BGP is no longer in conflicting point).
I see that there is support introduced for MetalLB in BGP mode with #354 (and #355). However, could I suggest adding support for the ARP mode as well?
I know that MetalLB's ARP mode has scaling issues - however I am in a scenario where the network is not controlled by me and does not support BGP. Therefore, could a flag be added that just treats LoadBalancer service IPs as if they were an externalIP field?
Obviously this will be less reliable/HA than the BGP method, but in some scenarios it's the only option.
@ccatlett2000 You should be able to use MetalLB ARP mode. kube-router does not advertise service's load balanacer IP's if not expllictly asked to do so i.e) by enabling --advertise-loadbalancer-ip flag.
Kube-router configures IPVS with load balancer IP. Please try and see.
cc @jjo
Indeed WfM, on my raspberry pi2 cluster fwiw.
Linking manifests for both (just note the arm image(s) for kube-router):
- kube-router
--advertise-loadbalancer-ips=true-> https://gist.github.com/jjo/8c616aaf795284bb5b85d02143745f63 - metallb w/ARP mode -> https://gist.github.com/jjo/bac1e4cd12c854fc58ca2de084b3b874
@murali-reddy Ah, I read this blog post and because of it I was under the impression that IPVS proxy wouldn't be created for LoadBalancer services. However, if that isn't the case I will certainly try it out when I set up this cluster today/tomorrow.
@ccatlett2000: you're correct, indeed --advertise-loadbalancer-ips was just recently added ( PR #354 ).
kube-router 0.2.0-beta.5 works perfect on 1.9.2 bare metal cluster with MetalLB and --advertise-loadbalancer-ips=true flag. All flags I am using are:
--kubeconfig=/etc/kubernetes/kube-router.conf --run-firewall=true --run-router=true --run-service-proxy=true --advertise-loadbalancer-ip
I have stopped using kube-proxy at all
@tiukhtinvladimir thanks for letting us know. We are pretty close to get 0.2.0 release out. Once released will update the documentation on kube-router and metalb compatibility.
/cc @danderson @jjo
Great news! Once 0.2.0 is released, I'll write documentation on the integration for MetalLB, and document which kube-router configurations are supported.
One clarification for me: if you set --advertise-loadbalancer-ips=true, what exactly happens? Does it do both dataplane programming on nodes, and advertising the service over BGP to external peers? Is the latter optional, i.e. if I don't want to peer with external peers, will you still program the node dataplane (i.e. same behavior as kube-proxy) ?
One clarification for me: if you set --advertise-loadbalancer-ips=true, what exactly happens? Does it do both dataplane programming on nodes, and advertising the service over BGP to external peers? Is the latter optional, i.e. if I don't want to peer with external peers, will you still program the node dataplane (i.e. same behavior as kube-proxy) ?
If --advertise-loadbalancer-ips is set, host node IP/IPVS ("dataplane programming") always happens, then BGP advertising if peers (--peers-... args) are set.
Is there any update on the documentation for using metallb w/ kube-router? @danderson
@murali-reddy Do you still have interest in supporting loadbalancer service types in kube-router now that metallb seems to be a reasonable alternative?
Maybe the conclusion to this ticket is just some some documentation on partnering kube-router with metallb?
@aauren a pair of kube-router and metallb works for me, but obviously I would prefer to have all in one, considering all required BGP functionality already implemented in kube-router
@murali-reddy Do you still have interest in supporting loadbalancer service types in kube-router now that metallb seems to be a reasonable alternative?
@aauren Yes. Except for IPAM (assigning the IP for the loadbalancer service from the pool of IP's) all functionality is already there in kube-router. Its just users started using kube-router along with metallb so this issue was not prioritized. However as mentioned having nativley in kube-router means one more less moving piece.
Maybe the conclusion to this ticket is just some some documentation on partnering kube-router with metallb?
We did that already. We have some documentation https://github.com/cloudnativelabs/kube-router/blob/master/docs/user-guide.md#advertising-ips
But somehow original documentation from https://github.com/cloudnativelabs/kube-router/pull/354 is lost