Sysbox Install - Pods stuck in ContainerCreating or Podinitializing state - cri-o configured with systemd cgroup manager, but did not receive slice as parent

[matthewparkinsondes]

Rancher 2.6.5 - RKE2 v1.23.6+rke2r1 - Sysbox-CE 0.5.2 - Ubuntu 20.04 - Kernel 5.13.0-39-generic - x86_64

An issue was observed when attempting to install Sysbox on an RKE2 kubernetes cluster. After the install, the pods in each worker node fail to come back online, and are stuck in the ContainerCreating or Podinitializing state.

The pods show the following error.

Failed to create pod sandbox: rpc error: code = Unknown desc = cri-o configured with systemd cgroup manager, but did not receive slice as parent: /kubepods/burstable/pod95c5e3b0-0305-4bd3-8cf7-019d19f9af0b

Below are logs from the sysbox-deploy-k8s-xxxxx pod.

Detected Kubernetes version v1.23 Adding K8s label "crio-runtime=installing" to node ... node/xxxxx labeled Deploying CRI-O installer agent on the host (v1.23) ... Running CRI-O installer agent on the host (may take several seconds) ... Removing CRI-O installer agent from the host ... Configuring CRI-O ... Adding K8s label "sysbox-runtime=installing" to node ... node/xxxxx labeled Installing Sysbox dependencies on host ... Copying shiftfs sources to host ... Kernel version 5.13 is >= 5.13 Deploying Sysbox installer helper on the host ... Running Sysbox installer helper on the host (may take several seconds) ... Stopping the Sysbox installer helper on the host ... Removing Sysbox installer helper from the host ... Installing Sysbox on host ... Created symlink from /etc/systemd/system/multi-user.target.wants/sysbox.service to /lib/systemd/system/sysbox.service. Created symlink from /etc/systemd/system/sysbox.service.wants/sysbox-mgr.service to /lib/systemd/system/sysbox-mgr.service. Created symlink from /etc/systemd/system/sysbox.service.wants/sysbox-fs.service to /lib/systemd/system/sysbox-fs.service. Configuring host sysctls ... kernel.unprivileged_userns_clone = 1 fs.inotify.max_queued_events = 1048576 fs.inotify.max_user_watches = 1048576 fs.inotify.max_user_instances = 1048576 kernel.keys.maxkeys = 20000 kernel.keys.maxbytes = 1400000 kernel.pid_max = 4194304 Starting Sysbox ... Adding Sysbox to CRI-O config ... Restarting CRI-O ... Deploying Kubelet config agent on the host ... Running Kubelet config agent on the host (will restart Kubelet and temporary bring down all pods on this node for ~1 min) ...

[FFock]

I have the same/simlar issue with Rancher 2.6.6, K8S v1.21.12, Ubuntu 20.04 (latest).

[FFock]

Same issue exists with Rancher 2.6.6, K8S v1.22.10, Ubuntu 20.04 (latest).

[dgadelha]

Reproduction steps:

1. Create 2 e2-medium VMs on GCP with Ubuntu 22.04 Minimal x86/64 (NOTE: Sysbox doesn't work with RKE control plane + worker in a single node [it doesn't recognize kubelet], so you need a separate worker-only node)
2. Go to a Rancher instance (we use v2.6.3), create a RKE2 cluster, no changes from the default options
3. Add one of the instances as control-plane + etcd + worker, without the label
4. Add the other instance as worker only, with the label sysbox-install=yes
5. Run: kubectl apply -f https://raw.githubusercontent.com/nestybox/sysbox/master/sysbox-k8s-manifests/sysbox-install.yaml
6. Install runs successfully, but after kubelet gets restarted, the issue reported in the first post happens, all pods in the node get stuck in ContainerCreating/PodInitializing

[rodnymolina]

@dgadelha, thanks for the repro steps. Will take a look at this one within the next few days.

[gdraganic]

@rodnymolina same issue on: RKE2 v1.23.10+rke2r1, sysbox-ce v0.5.2, Ubuntu 22.04, kernel 5.15.0-46-generic

When the RKE2 worker node is initially bootstrapped (with containerd), kubelet on the node is running with the following arg: --cgroup-driver=systemd

After successful sysbox installation kubelet is restarted and running with the --cgroup-driver=cgroupfs. However, sysbox generated crio configuration is missing the cgroup_manager setting which defaults to systemd (https://github.com/nestybox/cri-o/blob/main/docs/crio.conf.5.md)

Adding the following in /etc/crio/crio.conf:

[crio]
  ...
  [crio.runtime]
    conmon_cgroup = "pod"
    cgroup_manager = "cgroupfs"
  ...

and restarting crio with systemctl restart crio.service resolves the problem.

[rodnymolina]

@gdraganic, I think that's pretty much the issue here...

The sysbox daemon-set should not have switched the RKE2's kubelet to cgroupfs if RKE2 has originally picked systemd as its cgroup driver. The goal here is to reduce disruptions to the minimum, so we will need to check why this isn't happening.

In consequence, the current crio.conf file is displaying the proper configuration. The problem is with the kubelet one. We will look into this.

Btw, yes, you can also workaround the issue by doing what you suggested (switching crio to cgroupfs), but I think a better solution would be to do what I described above.

Thanks for your detailed analysis.

[brandond]

What exactly is sysbox doing here? Is it installing cri-o and then reconfiguring rke2 to use that as the --container-runtime-endpoint instead of the packaged containerd? RKE2 manages the cgroup driver config for both the kubelet and containerd if you use the packaged containerd, but if you bring your own container runtime, you're responsible for making sure that the kubelet's cgroup driver is the same as the container runtime. Sounds like sysbox is not doing that properly.

[ctalledo]

but if you bring your own container runtime, you're responsible for making sure that the kubelet's cgroup driver is the same as the container runtime.

That's correct @brandond; it * should * be doing that with Kubelet and CRI-O, but I'll let @rodnymolina double check it.

[TH3VenGeuR]

@rodnymolina any update ? Same issue on: Rancher 2.6.8, v1.23.8-rancher1, sysbox-ce v0.5.2, Ubuntu 20.04, kernel 5.4.0-1055-kvm