cloudnativelabs
How to route pod traffic via different network interfaces?
My Kubernetes nodes have a dedicated network interface that shall be used for the Kubernetes Pod network traffic (eth1) while all other traffic go for eth0. Also, the default gateway points to an IP address that is reachable via eth0. On eth1, a dedicated network is configured serving as transfer network for Pod traffic.
The setup looks like this:
- Node 1: Primary IP (eth0): 10.128.16.11, Pod transfer network IP (eth1): 10.128.18.11
- Node 2: Primary IP (eth0): 10.128.16.12, Pod transfer network IP (eth1): 10.128.18.12
- Node 3: Primary IP (eth0): 10.128.16.13, Pod transfer network IP (eth1): 10.128.18.13
- Pod CIDR: 10.128.32.0/20, to be routed via eth1
How can I configure kube-router to route Pod networks via eth1? I have not found any documentation and no suitable command line switch for this. Annotating the nodes with
kubectl annotate node node1 "kube-router.io/bgp-local-addresses=<pod_transfer_network_ip_on_eth1>"
only makes kube-router listening on this IP address for BGP sessions.
@stephan2012 please try https://github.com/cloudnativelabs/kube-router/blob/master/docs/bgp.md#overriding-the-next-hop
@murali-reddy, tried this without success. This feature seems to enable some automagic feature that I'm not sure how it works because all nodes are seeing each other over eth0 and eth1.
@janse180: Unfortunately I have not. In Calico you can pass the interface name but I have not found a similar option for kube-router. This makes it impossible to separate workload from management traffic.
Looking at the code it doesn't seem like this is possible right now. There are several locations where the Peer IP addresses are determined based on the NodeInternalIP or NodeExternalIP assigned by kubernetes. This makes it impossible for me to use this project as an alternative to calico on my bare metal cluster as the kubernetes control plane network is physically separate from my pod traffic network.
https://github.com/cloudnativelabs/kube-router/blob/46f8265e1b141cef258efcf2adb08aafddcafb76/pkg/utils/node.go#L43-L46
@stephan2012 can you please point me the documentation of calico.
Let me try to understand what is that missing in kube-router. @stephan2012 from your initial description if i make BGP server listen on just eth1, I would expect the routes and next hops for all the node CIDR's should have been all on eth1 network. Let me try to reproduce and see.
@murali-reddy: See https://docs.projectcalico.org/v3.3/reference/node/configuration#ip-autodetection-methods on how to configure a network interface for inter-Pod network traffic. Basically, it requires setting IP_AUTODETECTION_METHOD=interface=eth1. It requires a transfer network configured on the interfaces.
Just making the BGP peer listen on a different interface could redirect the Pod traffic through that interface. However, when using MetalLB this is not an option because this traffic might go through just another interface.
I created a fork that has a work around for this. It allows the user to override the IP address of the node that by default is pulled from the Kubernetes API. I'm currently using it on a bare-metal cluster with about 30 nodes and have not run into any issues.
https://github.com/janse180/kube-router/tree/nodeip-annotation
You can try it out by changing your container image to: janse180/kube-router-git:nodeip-annotation Then add an annotation to each node in your cluster with the IP of the interface you want that node to pass kube-router traffic on.
In your above case you would want to do the following:
kubectl annotate node node1 kube-router.io/nodeip=10.128.18.11
kubectl annotate node node2 kube-router.io/nodeip=10.128.18.12
kubectl annotate node node3 kube-router.io/nodeip=10.128.18.13
T
I created a fork that has a work around for this. It allows the user to override the IP address of the node that by default is pulled from the Kubernetes API. I'm currently using it on a bare-metal cluster with about 30 nodes and have not run into any issues.
https://github.com/janse180/kube-router/tree/nodeip-annotation
You can try it out by changing your container image to: janse180/kube-router-git:nodeip-annotation Then add an annotation to each node in your cluster with the IP of the interface you want that node to pass kube-router traffic on.
In your above case you would want to do the following:
kubectl annotate node node1 kube-router.io/nodeip=10.128.18.11 kubectl annotate node node2 kube-router.io/nodeip=10.128.18.12 kubectl annotate node node3 kube-router.io/nodeip=10.128.18.13
thanks very much for this , I also hit this issue. Will try your solution.
@murali-reddy do you plan to implement the changes provided by @janse180 to upstream?
This feature is really necessary to select preferred interface by kube-router on multiple interface nodes. Any idea when this will be possible with kube-router ?
I would like to see this feature as well. I have a fast admin network that I would like to use for pod traffic.
Unfortunately, at this time, the kube-router project is unable to consider more complex multi-homed use-cases beyond what it already supports.
