kube-router
kube-router copied to clipboard
Where appropriate kube-router should install multi-path routes
When multiple peer routers are configured, for example --peer-router-ips="192.168.0.201,192.168.0.202"
then router should be configured to all peer routers. This enables load sharing for outgoing traffic.
This is supported when gobgp uses the zebra backend.
All routers are in the RIB (narrowed);
# gobgp global rib
Network Next Hop AS_PATH Age Attrs
*> 192.168.2.0/24 192.168.0.201 65000 00:52:55 [{Origin: i} {Med: 0}]
* 192.168.2.0/24 192.168.0.202 65000 00:52:49 [{Origin: i} {Med: 0}]
But only one is actually used;
# ip ro
192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.2
192.168.2.0/24 via 192.168.0.201 dev eth0 proto 17
I use gobgp/zebra for test as GWs and multi-path routes are configured;
# ip ro
12.0.0.1 proto zebra metric 20
nexthop via 192.168.0.2 dev eth0 weight 1
nexthop via 192.168.0.3 dev eth0 weight 1
nexthop via 192.168.0.4 dev eth0 weight 1
nexthop via 192.168.0.1 dev eth0 weight 1
(12.0.0.1
is my kubernetes-ClusterIP, others are node addresses)
Adding to this it would be also good to mark any peer-router as fallback by a lower weigth. I.e. when the first route uses the private network and the second wents over the internet instead, you would not want it to route there unless the other routes are dead or saturated.
Actually, it's not required to use Zebra for that, just enable the MultiPath config option and then it will do.
But it's also required to update the route adding code to actually install multipath routes. (There might be an issue with gobgpd see: https://github.com/osrg/gobgp/issues/1958 )
any updates on this? this is really cool to have multi path beetween nodes in a cluster
I too think that east/west traffic, or simply any traffic originating or forwarded by kube-router being ECMP is pretty necessary given the amount of east/west bouncing around that goes on inside of a cluster.
For example my cluster is using a Clos fabric or 'leaf/spine' architecture so there are always at least 2 Equal Cost Paths between all of my nodes, and kube-router will only use one of them.
For now my solution is to also run BIRD on my nodes in-between KR and my spine routers. KR tells BIRD about the pod CIDRs & service IPs, and BIRD negotiates ECMP routes with the spine telling it about these destinations on the node. So I'm really looking forward to kube-router adopting this feature so I can retire BIRD.
Thanks for your hard work!
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.
This issue was closed because it has been stale for 5 days with no activity.