Add cross mesh granularity

[skirsten]

This is a re-implementation of https://github.com/squat/kilo/pull/98. A rebase was not possible so the code was rewritten but without any logic changes.

I deployed it in a test cluster running in addon + flannel mode:

--cni=false --compatibility=flannel --local=false --mesh-granularity=cross

The cluster has 4 nodes in 2 different subnets / regions. As can be seen by these example routes from one of the hosts this is working as expected:

$ ip route get 10.42.0.5
10.42.0.5 via 10.4.0.1 dev kilo0 src 10.4.0.3 uid 1000
$ ip route get 10.42.2.2
10.42.2.2 via 10.4.0.2 dev kilo0 src 10.4.0.3 uid 1000
$ ip route get 10.42.3.2
10.42.3.2 via 10.42.3.0 dev flannel.1 src 10.42.1.0 uid 1000
$ ip route get 10.42.1.3
10.42.1.3 dev cni0 src 10.42.1.1 uid 1000

One problem I found: If I upgrade a existing kilo deployment from full to cross granularity the flannel routing is broken until I restart the k3s agent. This might not be related with this code and rather that it was broken by the full granularity. Rebooting the instances also seems to work so this seems unrelated to me.

I will now implement the graph logic. If somebody can verify and test this in the CNI mode that would be great. I published a docker image here: ghcr.io/skirsten/kilo

[skirsten]

I also updated the graph logic to visualize the plain connection. Unfortunately the circo -Tsvg command (right) that is referred to here in the docs does not visualize the locations in a good way. On https://dreampuf.github.io/GraphvizOnline (left) it looks much better.

If somebody has a larger cluster with interesting topology and cross mesh granularity, they could add a cross.svg to the docs.

[skirsten]

I am now testing this in a 50 node cluster and its working great so far. But the kgctl graph is not working correctly. I will try to fix it.

[squat]

Amazing stuff @skirsten! I think this will be a great feature for Kilo. We'll need to add more tests before releasing, but that should not hold things up