Error: systemd not running on this host, cannot use systemd cgroups manager

[kgibm]

trafficstars

Running runc --root /host/run/runc list from an OpenShift debug pod shows errors:

First, starting the debug pod:

$ oc debug node/[...] -t --image=quay.io/kgibm/containerdiagsmall

Then running runc --root /host/run/runc list shows an error for each container:

sh-5.1# runc --root /host/run/runc list
load container 028ef8e5ac9c9a863e4570f5939e243ac5fe9f13a22286e395501cce2e6d0d31: systemd not running on this host, cannot use systemd cgroups manager
load container 03b6c1123e69ef08f5738f904b55f41960a7a675bcec2484c3e28d4f6cdb7c0d: systemd not running on this host, cannot use systemd cgroups manager
[...]
ID          PID         STATUS      BUNDLE      CREATED     OWNER

However, chrooting into the root and running its runc list works fine:

sh-5.1# chroot /host
sh-4.4# runc list
ID                                                                 PID         STATUS      BUNDLE                                                                                                                 CREATED                          OWNER
028ef8e5ac9c9a863e4570f5939e243ac5fe9f13a22286e395501cce2e6d0d31   2740        running     /run/containers/storage/overlay-containers/028ef8e5ac9c9a863e4570f5939e243ac5fe9f13a22286e395501cce2e6d0d31/userdata   2022-04-18T21:49:10.04013488Z    root
03b6c1123e69ef08f5738f904b55f41960a7a675bcec2484c3e28d4f6cdb7c0d   9230        running     /run/containers/storage/overlay-containers/03b6c1123e69ef08f5738f904b55f41960a7a675bcec2484c3e28d4f6cdb7c0d/userdata   2022-04-18T21:50:36.952063647Z   root
[...]

Is this expected?

The runc in the debug pod image is installed on top of a Fedora 35 image: https://github.com/kgibm/dockerdebug/blob/master/containerdiagsmall/Containerfile#L37

[kom0055]

I got the same problem, case when I try to use runc/libcontainer/cgroupmanager in container. Is there any chance to read file from environment, not just hard-coding "/run/systemd/system"

[kgibm]

Running the host's runc shows the same issue:

# /host/usr/bin/runc --root /host/run/runc list
load container 028ef8e5ac9c9a863e4570f5939e243ac5fe9f13a22286e395501cce2e6d0d31: systemd not running on this host, cannot use systemd cgroups manager
[...]

The following workaround works:

# chroot /host runc list
ID                                                                 PID         STATUS      BUNDLE                                                                                                                 CREATED                          OWNER
028ef8e5ac9c9a863e4570f5939e243ac5fe9f13a22286e395501cce2e6d0d31   2740        running     /run/containers/storage/overlay-containers/028ef8e5ac9c9a863e4570f5939e243ac5fe9f13a22286e395501cce2e6d0d31/userdata   2022-04-18T21:49:10.04013488Z    root
[...]

[kolyshkin]

Hmm. This can be fixed, although it's going to be complicated. Thanks for reporting, I'll look into that.

[kolyshkin]

Is there any chance to read file from environment

Sure. Is there an env var available that can be used?

[kolyshkin]

Is there any chance to read file from environment, not just hard-coding "/run/systemd/system"

Also, a (somewhat nasty) workaround would be to do mkdir /run/systemd/system.

[kom0055]

Also, a (somewhat nasty) workaround would be to do mkdir /run/systemd/system.

Cause I run this in a container, I cannot use it unless I mount /run/systemd readonly on my container, but I think it's not that elegant and lack of safety, it could be /rootfs/run/systemd cause I already mounted the root dir on /rootfs. Or we could think in a by pass way, like github.com/godbus/dbus/v5/conn_unix.go:12 , this module support we assgin the dbus address from environment, use env variable DBUS_SYSTEM_BUS_ADDRESS

[vipulnewaskar7]

Is there any chance to read file from environment

Sure. Is there an env var available that can be used?

Hi @kolyshkin Not sure if this will help... If somehow we get the RootDirectory path of "chrooted" host ("/host" in this case) from user, then we can use systemd executable of the host itself. https://www.freedesktop.org/software/systemd/man/systemd.exec.html#RootDirectory=

[kolyshkin]

Sure. Is there an env var available that can be used?

Hi @kolyshkin Not sure if this will help... If somehow we get the RootDirectory path of "chrooted" host ("/host" in this case) from user, then we can use systemd executable of the host itself. https://www.freedesktop.org/software/systemd/man/systemd.exec.html#RootDirectory=

This does not look like an environment variable. It's a systemd unit configuration parameter.

Or we could think in a by pass way, like github.com/godbus/dbus/v5/conn_unix.go:12 , this module support we assgin the dbus address from environment, use env variable DBUS_SYSTEM_BUS_ADDRESS

We can actually add an alternative method to see if systemd is running -- by trying to connect to it. I'll take a look

[kolyshkin]

OK, this seems complicated because of how we initialize dbus connection depending on whether runc is rootless or not.

I'm a bit stuck here. Looks like the alternative (not looking into systemd when listing containers) is a better idea.

[vipulnewaskar7]

runc list well as other methods (indirectly) call libcontainer.Load(root, item.Name()) so we might need to handle those scenarios also? otherwise, issues will appear for commands other than the list command

https://github.com/opencontainers/runc/blob/9524366183517b379a1fca8b9b250002ba3b1be7/list.go#L112-L144

https://github.com/opencontainers/runc/blob/9524366183517b379a1fca8b9b250002ba3b1be7/utils_linux.go#L28-L34

[kgibm]

Just to add that my workaround of chroot /host runc list does not always work. It works fine with oc but not with kubectl (targeting the same cluster):

$ kubectl debug node/$NODE -it --image=$IMAGE -- bash
[...]
[root@worker2 /]# chroot /host runc list
chroot: cannot change root directory to '/host': Operation not permitted

Yet /host does exist and I'm root:

[root@worker2 /]# ls /host
bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  ostree  proc  root  run  sbin  srv  sys  sysroot  tmp  usr  var
[root@worker2 /]# id
uid=0(root) gid=0(root) groups=0(root) context=system_u:system_r:spc_t:s0

Maybe kubectl uses less privileged mount flags?

# mount | grep '/host '
/dev/vda4 on /host type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota)

In any case, a workaround using unshare works:

[root@worker2 /]# unshare -rR /host runc list
ID                                                                 PID         STATUS      BUNDLE                                                                                                                 CREATED                          OWNER
01c8656304e4043be24fb73a932e5688d2f49f32f451741c6b5c60ce830ded68[...]

Kubectl version:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.6", GitCommit:"ad3338546da947756e8a88aa6822e9c11e7eac22", GitTreeState:"clean", BuildDate:"2022-04-14T08:41:58Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"darwin/amd64"} [...]

Comparing mount flags and id with oc shows no differences so not sure what's preventing the chroot with kubectl:

[root@worker2 /]# mount | grep '/host '
/dev/vda4 on /host type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota)
[root@worker2 /]# id
uid=0(root) gid=0(root) groups=0(root) context=system_u:system_r:spc_t:s0