Kind: kindnet crashing right after Liqo installation

[NominalTrajectory]

Originally posted by @NominalTrajectory in https://github.com/liqotech/liqo/issues/314#issuecomment-1029951297

Hi there! I think I'm experiencing the exact same issue today testing Liqo.

I'm following a quickstart guide described here https://doc.liqo.io/gettingstarted/helloworld/ and using the exact same configurations provided using kind v0.11.1 go1.16.4 darwin/amd64 with Docker Desktop on Mac.

Liqo installs successfully, but as soon as the virtual-kubelet gets created, I see crashing kindnet and kube-proxy pods.

My kindnet logs are:

I0204 12:36:30.478346       1 main.go:316] probe TCP address cluster1-control-plane:6443
I0204 12:36:30.480330       1 main.go:102] connected to apiserver: https://cluster1-control-plane:6443
I0204 12:36:30.480369       1 main.go:107] hostIP = 172.19.0.3
podIP = 172.19.0.3
I0204 12:36:30.480601       1 main.go:116] setting mtu 1500 for CNI
I0204 12:36:30.480640       1 main.go:146] kindnetd IP family: "ipv4"
I0204 12:36:30.480666       1 main.go:150] noMask IPv4 subnets: [10.200.0.0/16]
I0204 12:36:31.075989       1 main.go:223] Handling node with IPs: map[172.19.0.3:{}]
I0204 12:36:31.076049       1 main.go:227] handling current node
I0204 12:36:31.078393       1 main.go:223] Handling node with IPs: map[172.19.0.2:{}]
I0204 12:36:31.078506       1 main.go:250] Node cluster1-worker has CIDR [10.200.1.0/24]
I0204 12:36:31.078776       1 main.go:223] Handling node with IPs: map[10.200.1.9:{}]
I0204 12:36:31.078816       1 main.go:250] Node liqo-b761b1b3-d06c-46a3-94a0-a025c0d2f147 has CIDR [10.200.2.0/24]
I0204 12:36:31.078941       1 routes.go:46] Adding route {Ifindex: 0 Dst: 10.200.2.0/24 Src: <nil> Gw: 10.200.1.9 Flags: [] Table: 0}
I0204 12:36:31.079222       1 main.go:204] Failed to reconcile routes, retrying after error: network is unreachable
I0204 12:36:31.079427       1 main.go:223] Handling node with IPs: map[172.19.0.3:{}]
I0204 12:36:31.079493       1 main.go:227] handling current node
I0204 12:36:31.079511       1 main.go:223] Handling node with IPs: map[172.19.0.2:{}]
I0204 12:36:31.079762       1 main.go:250] Node cluster1-worker has CIDR [10.200.1.0/24]
I0204 12:36:31.080011       1 main.go:223] Handling node with IPs: map[10.200.1.9:{}]
I0204 12:36:31.080094       1 main.go:250] Node liqo-b761b1b3-d06c-46a3-94a0-a025c0d2f147 has CIDR [10.200.2.0/24]
I0204 12:36:31.080244       1 routes.go:46] Adding route {Ifindex: 0 Dst: 10.200.2.0/24 Src: <nil> Gw: 10.200.1.9 Flags: [] Table: 0}
I0204 12:36:31.080403       1 main.go:204] Failed to reconcile routes, retrying after error: network is unreachable
I0204 12:36:32.080503       1 main.go:223] Handling node with IPs: map[172.19.0.3:{}]
I0204 12:36:32.080603       1 main.go:227] handling current node
I0204 12:36:32.080628       1 main.go:223] Handling node with IPs: map[172.19.0.2:{}]
I0204 12:36:32.080651       1 main.go:250] Node cluster1-worker has CIDR [10.200.1.0/24]
I0204 12:36:32.080802       1 main.go:223] Handling node with IPs: map[10.200.1.9:{}]
I0204 12:36:32.080827       1 main.go:250] Node liqo-b761b1b3-d06c-46a3-94a0-a025c0d2f147 has CIDR [10.200.2.0/24]
I0204 12:36:32.081020       1 routes.go:46] Adding route {Ifindex: 0 Dst: 10.200.2.0/24 Src: <nil> Gw: 10.200.1.9 Flags: [] Table: 0}
I0204 12:36:32.081097       1 main.go:204] Failed to reconcile routes, retrying after error: network is unreachable
I0204 12:36:34.081567       1 main.go:223] Handling node with IPs: map[172.19.0.3:{}]
I0204 12:36:34.081681       1 main.go:227] handling current node
I0204 12:36:34.081715       1 main.go:223] Handling node with IPs: map[172.19.0.2:{}]
I0204 12:36:34.081740       1 main.go:250] Node cluster1-worker has CIDR [10.200.1.0/24]
I0204 12:36:34.081976       1 main.go:223] Handling node with IPs: map[10.200.1.9:{}]
I0204 12:36:34.082001       1 main.go:250] Node liqo-b761b1b3-d06c-46a3-94a0-a025c0d2f147 has CIDR [10.200.2.0/24]
I0204 12:36:34.082069       1 routes.go:46] Adding route {Ifindex: 0 Dst: 10.200.2.0/24 Src: <nil> Gw: 10.200.1.9 Flags: [] Table: 0}
I0204 12:36:34.082150       1 main.go:204] Failed to reconcile routes, retrying after error: network is unreachable
I0204 12:36:37.082937       1 main.go:223] Handling node with IPs: map[172.19.0.3:{}]
I0204 12:36:37.083409       1 main.go:227] handling current node
I0204 12:36:37.083493       1 main.go:223] Handling node with IPs: map[172.19.0.2:{}]
I0204 12:36:37.083546       1 main.go:250] Node cluster1-worker has CIDR [10.200.1.0/24]
I0204 12:36:37.083698       1 main.go:223] Handling node with IPs: map[10.200.1.9:{}]
I0204 12:36:37.083771       1 main.go:250] Node liqo-b761b1b3-d06c-46a3-94a0-a025c0d2f147 has CIDR [10.200.2.0/24]
I0204 12:36:37.083964       1 routes.go:46] Adding route {Ifindex: 0 Dst: 10.200.2.0/24 Src: <nil> Gw: 10.200.1.9 Flags: [] Table: 0}
I0204 12:36:37.084140       1 main.go:204] Failed to reconcile routes, retrying after error: network is unreachable
panic: Maximum retries reconciling node routes: network is unreachable

goroutine 1 [running]:
main.main()
	/go/src/cmd/kindnetd/main.go:208 +0x1327

To solve this problem for KindNet it would be enough to remove the PodCIDR from the node specs of the virtual node.

I tried this but it didn't seem to work for me.

@palexster can this issue be reopened?

[alacuku]

Hi @NominalTrajectory,

this is a known problem. The KIND CNI is a simple one that tries to configure the routes for all the nodes. When it fails to do so, it panics. Kindnet going in crashloopbackoff does not mean that Liqo does not work in KIND as long as you use it only for demo purposes. Looking at the code of kindnet, the only workaround is to remove the podCIDR from the virtual node created by Liqo: https://github.com/kubernetes-sigs/kind/blob/80a6a355a2065823bbdae5048abf76a4f2a3ec9d/images/kindnetd/cmd/kindnetd/main.go#L246.