terraform-hcloud-kube-hetzner
terraform-hcloud-kube-hetzner copied to clipboard
kube-hetzner
Cilium failing after automatic reboot
Hi guys, great work implementing Cilium as a CNI option in this awesome project! 🙏 While testing out Cilium with this project I've noticed that after node reboots through e.g. kured many pods become unmanaged by Cilium and stop working correctly as a result. Would we want to set a Cilium specific taint per default when CNI = cilium as mentioned in the Cilium docs to prevent the aforementioned from happening? I'll have time to try out the mentioned taint later today or tomorrow and am happy to submit a short PR for you to review if everything works. Just wanted to reach out and give a heads up 👍 FYI: @mysticaltech , @phaer , @PurpleBooth
Originally posted by @t1murl in https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/issues/258#issuecomment-1207372796
Hi @mysticaltech , replying here - thanks for opening this issue! My week is unfortunately quite busy with my day job. I usually find time at the weekends, mostly Saturdays. I can therefore offer time on Saturday at the latest to fully analyze this - however, I had figured you'd want to address this earlier hence giving the initial heads up.
The mechanism works as follows:
The cluster administrator places a specific taint (see below) on a given uninitialized node. Depending on the taint’s effect (see below), this prevents pods that don’t have a matching toleration from either being scheduled or altogether running on the node until the taint is removed.
Cilium runs on the node, initializes it and, once ready, removes the aforementioned taint.
From this point on, pods will start being scheduled and running on the node, having their networking managed by Cilium.
Taking this quote from the Cilium docs into consideration, I'd go about it like such (without having tried it):
- Taint all nodes with 'node.cilium.io/agent-not-ready=true:NoExecute' per default
- Bootstrap Cilium before anything else
- Wait for Cilium to remove the taint via looping over e.g.
kubectl get nodes -o json | jq '.items[].spec.taints'checking for the existence of the taint - Deploy the remaining kustomizations
From quickly scanning through the way how taints are applied through the k3s config, I suppose the taint is re-applied after a node reboot (which is a good thing) and should be removed again by Cilium once fully initialized, so the aforementioned steps should only be required for the first time kustomization apply. This is something that would need to be tested thoroughly to avoid unwanted side effects.
I hope this helps and sorry for not being able to offer a contribution right away. (Would've been different if I would've started looking at this on Saturday 😞 )
This helps a bunch @t1murl, thanks a lot for taking the time to recap everything so clearly 🙏
Folks, for transparency, I was too ambitious; I probably won't have the time for this for the next two weeks! 🥺 @t1murl, please don't hesitate; PR is more than welcome. I will check Github daily.