cloudnativelabs
source address used when sending packets over tunnel interface is wrong
In multi-homed nodes node's can have multiple IP's. It is observed that source address used when sending the packets is not exaclty the node IP.
In an e.g setup
m01 Ready master 1d v1.11.0 10.20.0.100 <none> Debian GNU/Linux 9 (stretch) 4.9.0-6-amd64 docker://17.3.2
s01 Ready <none> 1d v1.11.0 10.20.0.101 <none> Debian GNU/Linux 9 (stretch) 4.9.0-6-amd64 docker://17.3.2
s02 Ready <none> 18h v1.11.0 10.20.0.102 <none> Debian GNU/Linux 9 (stretch) 4.9.0-6-amd64 docker://17.3.2
s03 Ready <none> 18h v1.11.0 10.20.0.103 <none> Debian GNU/Linux 9 (stretch) 4.9.0-6-amd64 docker://17.3.2
Tunnel interfaces are established as
11: tun-10200103@wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1380 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1
link/ipip 10.20.0.100 peer 10.20.0.103
12: tun-10200102@wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1380 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1
link/ipip 10.20.0.100 peer 10.20.0.102
13: tun-10200101@wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1380 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1
link/ipip 10.20.0.100 peer 10.20.0.101
IP route is setup to use source IP which is the node IP as well
10.244.1.0/24 dev tun-10200101 proto 17 src 10.20.0.100
10.244.2.0/24 dev tun-10200103 proto 17 src 10.20.0.100
10.244.3.0/24 dev tun-10200102 proto 17 src 10.20.0.100
However when packet is sent on the tunnel interface, in case of multi-homed network IP used is not the node ip.
root@m01 ~ # tcpdump -n -i tun-10200103
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tun-10200103, link-type RAW (Raw IP), capture size 262144 bytes
19:06:13.268478 IP 159.69.49.195.50250 > 10.244.2.16.8080: Flags [S], seq 621191744, win 43690, options [mss 65495,sackOK,TS val 851435 ecr 0,nop,wscale 7], length 0
19:06:14.278700 IP 159.69.49.195.50250 > 10.244.2.16.8080: Flags [S], seq 621191744, win 43690, options [mss 65495,sackOK,TS val 851688 ecr 0,nop,wscale 7], length 0
AS you see source IP is 159.69.49.195 instead of 10.20.0.100.
For kube-router to function correctly correct source IP addressed should be used.
@murali-reddy : I still did not find a way make the service trafic route via wireguard. I've switched to calico and things work so it means they are doing something good. Let me know if I can provide any information but right now I won't have time to invesitgate kube-rotuer for some time.
