long-term rkt replacement

[databus23]

With rkt in the bin we need a long-term solution on how to run the kubelet (and other pods/containers on evert node).

In general we need a solution for the kubelet, everything else could be (and probably should be) run in daemonsets. We have a WIP to do that: https://github.com/sapcc/kubernikus/pull/566 Running daemonset in the clusters requires a reconciler that does the right thing when stuff gets modified or the cluster is upgraded.

Possible options:

• run the kubelet natively on the flatcar node
  • seems to be the recommended (kubeadmin style) approach, has the risk of incompatibilities with certain binaries that the kubelet wants to use
• run it using systemd-nspawn somewhat similar to how rkt did it
  • no prior art found. sounds nice but devil could be in the details. No experience with low level pluming like systemd-nspawnd in the team
• run it using docker/containerd
• daemon based container runtimes for kubelet are a little overkill and also mix up running kubelet with the container runtime also used by the kubelet for pods.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jknipper]

Is solved in support for k8s versions >=1.24.