kube-router icon indicating copy to clipboard operation
kube-router copied to clipboard

Where appropriate kube-router should install multi-path routes

Open uablrek opened this issue 6 years ago • 4 comments

When multiple peer routers are configured, for example --peer-router-ips="192.168.0.201,192.168.0.202" then router should be configured to all peer routers. This enables load sharing for outgoing traffic.

This is supported when gobgp uses the zebra backend.

All routers are in the RIB (narrowed);

# gobgp global rib
   Network              Next Hop             AS_PATH              Age        Attrs
*> 192.168.2.0/24       192.168.0.201        65000                00:52:55   [{Origin: i} {Med: 0}]
*  192.168.2.0/24       192.168.0.202        65000                00:52:49   [{Origin: i} {Med: 0}]

But only one is actually used;

# ip ro
192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.2 
192.168.2.0/24 via 192.168.0.201 dev eth0 proto 17 

I use gobgp/zebra for test as GWs and multi-path routes are configured;

# ip ro
12.0.0.1 proto zebra metric 20 
        nexthop via 192.168.0.2  dev eth0 weight 1
        nexthop via 192.168.0.3  dev eth0 weight 1
        nexthop via 192.168.0.4  dev eth0 weight 1
        nexthop via 192.168.0.1  dev eth0 weight 1

(12.0.0.1 is my kubernetes-ClusterIP, others are node addresses)

uablrek avatar Apr 24 '18 14:04 uablrek

Adding to this it would be also good to mark any peer-router as fallback by a lower weigth. I.e. when the first route uses the private network and the second wents over the internet instead, you would not want it to route there unless the other routes are dead or saturated.

wzrdtales avatar Aug 30 '18 18:08 wzrdtales

Actually, it's not required to use Zebra for that, just enable the MultiPath config option and then it will do.

But it's also required to update the route adding code to actually install multipath routes. (There might be an issue with gobgpd see: https://github.com/osrg/gobgp/issues/1958 )

thoro avatar May 06 '19 09:05 thoro

any updates on this? this is really cool to have multi path beetween nodes in a cluster

varuzam avatar Aug 13 '21 17:08 varuzam

I too think that east/west traffic, or simply any traffic originating or forwarded by kube-router being ECMP is pretty necessary given the amount of east/west bouncing around that goes on inside of a cluster.

For example my cluster is using a Clos fabric or 'leaf/spine' architecture so there are always at least 2 Equal Cost Paths between all of my nodes, and kube-router will only use one of them.

For now my solution is to also run BIRD on my nodes in-between KR and my spine routers. KR tells BIRD about the pod CIDRs & service IPs, and BIRD negotiates ECMP routes with the spine telling it about these destinations on the node. So I'm really looking forward to kube-router adopting this feature so I can retire BIRD.

Thanks for your hard work!

3rett avatar Feb 01 '22 19:02 3rett

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

github-actions[bot] avatar Sep 06 '23 02:09 github-actions[bot]

This issue was closed because it has been stale for 5 days with no activity.

github-actions[bot] avatar Sep 11 '23 02:09 github-actions[bot]