nri icon indicating copy to clipboard operation
nri copied to clipboard
verified publisher icon containerd

plugin helpers: A function to get the absolute path to the CGroupsV2

Open ngopalak-redhat opened this issue 3 months ago • 9 comments

Fixes: #35

Added a function to get the absolute path for cgroups. It will fall back to /sys/fs/cgroup in case there are issues accessing /proc/mounts, where the cgroups mount is defined.

Added a flag for the nri-logger to print the Cgroup path so that it has minimal impact on the plugin. Consider it an example for developers who need this function.

Examples

	// Examples:
	// 1. "kubepods-besteffort-pod123.slice/crio:container456" (container)
	//    -> "/sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod123.slice/crio-container456.scope"
	// 2. "kubepods-besteffort-pod123.slice:crio:container456" (container)
	//    -> "/sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod123.slice/crio-container456.scope"
	// 3. "kubepods-besteffort-pod123.slice" (pod)
	//    -> "/sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod123.slice"

Here's an example for expansion used by expandSliceHierarchy() method:

// Example: "kubepods-besteffort-pod123.slice" -> ["kubepods.slice", "kubepods-besteffort.slice", "kubepods-besteffort-pod123.slice"]

Special cases

  • "cri-o" for the sandbox pod does not suffix ".scope" to the cgroups path. "container-d" uses ".scope" suffix for all the paths. Hence "cri-o" is handled as a special case.

Test Process

  • Enable the logger plugin with enable-cgroups-log flag
  • Create a nginx pod
  • Get the logs from the NRI daemonset pod printing the absolute and relative paths
  • ssh/exec into the node and do "ls <abs_dir_name>"

Repeat for cri-o and container-d

Testing with cri-o on openshift cluster:

oc logs nri-plugin-logger-n49j8 -n kube-system | grep relativePath
time="2025-09-02T06:45:00Z" level=info msg="PodSandbox CGroups" absolutePath=/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35550f44_f691_427e_b57d_ab6127d2cb65.slice/crio-468f848e2ebec7203ed55748d08337851af589ac42eb551c572446a5c68fe631 relativePath="kubepods-burstable-pod35550f44_f691_427e_b57d_ab6127d2cb65.slice:crio:468f848e2ebec7203ed55748d08337851af589ac42eb551c572446a5c68fe631"
time="2025-09-02T06:45:00Z" level=info msg="Container CGroups" absolutePath=/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35550f44_f691_427e_b57d_ab6127d2cb65.slice/crio-805b902ed98fdc1409c76e9aa94964787a904c80f46fc3ee74283509108f5e73.scope relativePath="kubepods-burstable-pod35550f44_f691_427e_b57d_ab6127d2cb65.slice:crio:805b902ed98fdc1409c76e9aa94964787a904c80f46fc3ee74283509108f5e73"
time="2025-09-02T06:45:26Z" level=info msg="PodSandbox CGroups" absolutePath=/sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38186edb_96bd_4383_baf7_3b4edf3b5937.slice/crio-0f2b016bbc95b352c2b4287b3b654cb234b95efdeea1b35a170a6ed572ac2692 relativePath="kubepods-besteffort-pod38186edb_96bd_4383_baf7_3b4edf3b5937.slice:crio:0f2b016bbc95b352c2b4287b3b654cb234b95efdeea1b35a170a6ed572ac2692"
time="2025-09-02T06:45:26Z" level=info msg="Container CGroups" absolutePath=/sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38186edb_96bd_4383_baf7_3b4edf3b5937.slice/crio-248f581aef21ff1ea0d22051b3b4c19782ef907a19872f7ac5b599cb63859b38.scope relativePath="kubepods-besteffort-pod38186edb_96bd_4383_baf7_3b4edf3b5937.slice:crio:248f581aef21ff1ea0d22051b3b4c19782ef907a19872f7ac5b599cb63859b38"
core@ngopalak-ubuntu:~/go/src/github.com/ngopalak-redhat/nri$ oc debug node/ip-10-0-74-142.ec2.internal
Starting pod/ip-10-0-74-142ec2internal-debug-kt9cx ...
To use host binaries, run `chroot /host`. Instead, if you need to access host namespaces, run `nsenter -a -t 1`.
Pod IP: 10.0.74.142
If you don't see a command prompt, try pressing enter.
sh-5.1# chroot /host
sh-5.1# ls /sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38186edb_96bd_4383_baf7_3b4edf3b5937.slice/crio-0f2b016bbc95b352c2b4287b3b654cb234b95efdeea1b35a170a6ed572ac2692
cgroup.controllers	cpu.max.burst			 hugetlb.1GB.events	   io.bfq.weight	memory.oom.group	misc.events
cgroup.events		cpu.stat			 hugetlb.1GB.events.local  io.latency		memory.peak		misc.events.local
cgroup.freeze		cpu.stat.local			 hugetlb.1GB.max	   io.max		memory.reclaim		misc.max
cgroup.kill		cpu.weight			 hugetlb.1GB.numa_stat	   io.stat		memory.stat		misc.peak
cgroup.max.depth	cpu.weight.nice			 hugetlb.1GB.rsvd.current  io.weight		memory.swap.current	pids.current
cgroup.max.descendants	cpuset.cpus			 hugetlb.1GB.rsvd.max	   memory.current	memory.swap.events	pids.events
cgroup.procs		cpuset.cpus.effective		 hugetlb.2MB.current	   memory.events	memory.swap.high	pids.events.local
cgroup.stat		cpuset.cpus.exclusive		 hugetlb.2MB.events	   memory.events.local	memory.swap.max		pids.max
cgroup.subtree_control	cpuset.cpus.exclusive.effective  hugetlb.2MB.events.local  memory.high		memory.swap.peak	pids.peak
cgroup.threads		cpuset.cpus.partition		 hugetlb.2MB.max	   memory.low		memory.zswap.current	rdma.current
cgroup.type		cpuset.mems			 hugetlb.2MB.numa_stat	   memory.max		memory.zswap.max	rdma.max
cpu.idle		cpuset.mems.effective		 hugetlb.2MB.rsvd.current  memory.min		memory.zswap.writeback
cpu.max			hugetlb.1GB.current		 hugetlb.2MB.rsvd.max	   memory.numa_stat	misc.current
sh-5.1# ls /sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38186edb_96bd_4383_baf7_3b4edf3b5937.slice/crio-248f581aef21ff1ea0d22051b3b4c19782ef907a19872f7ac5b599cb63859b38.scope
cgroup.controllers	cpu.max.burst			 hugetlb.1GB.events	   io.bfq.weight	memory.oom.group	misc.events
cgroup.events		cpu.stat			 hugetlb.1GB.events.local  io.latency		memory.peak		misc.events.local
cgroup.freeze		cpu.stat.local			 hugetlb.1GB.max	   io.max		memory.reclaim		misc.max
cgroup.kill		cpu.weight			 hugetlb.1GB.numa_stat	   io.stat		memory.stat		misc.peak
cgroup.max.depth	cpu.weight.nice			 hugetlb.1GB.rsvd.current  io.weight		memory.swap.current	pids.current
cgroup.max.descendants	cpuset.cpus			 hugetlb.1GB.rsvd.max	   memory.current	memory.swap.events	pids.events
cgroup.procs		cpuset.cpus.effective		 hugetlb.2MB.current	   memory.events	memory.swap.high	pids.events.local
cgroup.stat		cpuset.cpus.exclusive		 hugetlb.2MB.events	   memory.events.local	memory.swap.max		pids.max
cgroup.subtree_control	cpuset.cpus.exclusive.effective  hugetlb.2MB.events.local  memory.high		memory.swap.peak	pids.peak
cgroup.threads		cpuset.cpus.partition		 hugetlb.2MB.max	   memory.low		memory.zswap.current	rdma.current
cgroup.type		cpuset.mems			 hugetlb.2MB.numa_stat	   memory.max		memory.zswap.max	rdma.max
cpu.idle		cpuset.mems.effective		 hugetlb.2MB.rsvd.current  memory.min		memory.zswap.writeback
cpu.max			hugetlb.1GB.current		 hugetlb.2MB.rsvd.max	   memory.numa_stat	misc.current
sh-5.1#

Test result: I can see ls working for the path given in the log

Testing with containerd on kind:

kubectl logs nri-plugin-logger-spsjc -n kube-system | grep relativePath
time="2025-09-02T06:40:23Z" level=info msg="PodSandbox CGroups" absolutePath=/sys/fs/cgroup/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-besteffort.slice/kubelet-kubepods-besteffort-pod1e597579_417e_482f_bf65_5fe170c3aebd.slice/cri-containerd-d7dfb4a21fe1703a8537e60d5af248af729315d187b24a538102d14874c8117d.scope relativePath="kubelet-kubepods-besteffort-pod1e597579_417e_482f_bf65_5fe170c3aebd.slice:cri-containerd:d7dfb4a21fe1703a8537e60d5af248af729315d187b24a538102d14874c8117d"
time="2025-09-02T06:40:25Z" level=info msg="Container CGroups" absolutePath=/sys/fs/cgroup/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-besteffort.slice/kubelet-kubepods-besteffort-pod1e597579_417e_482f_bf65_5fe170c3aebd.slice/cri-containerd-a09bc17f5aa29b0dbdec9ede5f46cf1396d2405917ab02e6f1eb1019f388396e.scope relativePath="kubelet-kubepods-besteffort-pod1e597579_417e_482f_bf65_5fe170c3aebd.slice:cri-containerd:a09bc17f5aa29b0dbdec9ede5f46cf1396d2405917ab02e6f1eb1019f388396e"
core@ngopalak-kind:~/nri$ docker exec -it kind-worker /bin/bash
root@kind-worker:/# ls /sys/fs/cgroup/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-besteffort.slice/kubelet-kubepods-besteffort-pod1e597579_417e_482f_bf65_5fe170c3aebd.slice/cri-containerd-d7dfb4a21fe1703a8537e60d5af248af729315d187b24a538102d14874c8117d.scope
cgroup.controllers	cgroup.threads	 cpuset.cpus.effective	   hugetlb.1GB.rsvd.max      io.stat		  memory.oom.group     misc.events
cgroup.events		cgroup.type	 cpuset.cpus.partition	   hugetlb.2MB.current	     io.weight		  memory.peak	       misc.max
cgroup.freeze		cpu.idle	 cpuset.mems		   hugetlb.2MB.events	     memory.current	  memory.pressure      pids.current
cgroup.kill		cpu.max		 cpuset.mems.effective	   hugetlb.2MB.events.local  memory.events	  memory.reclaim       pids.events
cgroup.max.depth	cpu.max.burst	 hugetlb.1GB.current	   hugetlb.2MB.max	     memory.events.local  memory.stat	       pids.max
cgroup.max.descendants	cpu.pressure	 hugetlb.1GB.events	   hugetlb.2MB.numa_stat     memory.high	  memory.swap.current  pids.peak
cgroup.pressure		cpu.stat	 hugetlb.1GB.events.local  hugetlb.2MB.rsvd.current  memory.low		  memory.swap.events   rdma.current
cgroup.procs		cpu.weight	 hugetlb.1GB.max	   hugetlb.2MB.rsvd.max      memory.max		  memory.swap.high     rdma.max
cgroup.stat		cpu.weight.nice  hugetlb.1GB.numa_stat	   io.max		     memory.min		  memory.swap.max
cgroup.subtree_control	cpuset.cpus	 hugetlb.1GB.rsvd.current  io.pressure		     memory.numa_stat	  misc.current
root@kind-worker:/# ls /sys/fs/cgroup/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-besteffort.slice/kubelet-kubepods-besteffort-pod1e597579_417e_482f_bf65_5fe170c3aebd.slice/cri-containerd-a09bc17f5aa29b0dbdec9ede5f46cf1396d2405917ab02e6f1eb1019f388396e.scope
cgroup.controllers	cgroup.threads	 cpuset.cpus.effective	   hugetlb.1GB.rsvd.max      io.stat		  memory.oom.group     misc.events
cgroup.events		cgroup.type	 cpuset.cpus.partition	   hugetlb.2MB.current	     io.weight		  memory.peak	       misc.max
cgroup.freeze		cpu.idle	 cpuset.mems		   hugetlb.2MB.events	     memory.current	  memory.pressure      pids.current
cgroup.kill		cpu.max		 cpuset.mems.effective	   hugetlb.2MB.events.local  memory.events	  memory.reclaim       pids.events
cgroup.max.depth	cpu.max.burst	 hugetlb.1GB.current	   hugetlb.2MB.max	     memory.events.local  memory.stat	       pids.max
cgroup.max.descendants	cpu.pressure	 hugetlb.1GB.events	   hugetlb.2MB.numa_stat     memory.high	  memory.swap.current  pids.peak
cgroup.pressure		cpu.stat	 hugetlb.1GB.events.local  hugetlb.2MB.rsvd.current  memory.low		  memory.swap.events   rdma.current
cgroup.procs		cpu.weight	 hugetlb.1GB.max	   hugetlb.2MB.rsvd.max      memory.max		  memory.swap.high     rdma.max
cgroup.stat		cpu.weight.nice  hugetlb.1GB.numa_stat	   io.max		     memory.min		  memory.swap.max
cgroup.subtree_control	cpuset.cpus	 hugetlb.1GB.rsvd.current  io.pressure		     memory.numa_stat	  misc.current

Test result: I can see ls working for the path given in the log

Test with cgroupfs on containerd with kind

core@ngopalak-kind:~/nri$ kubectl logs nri-plugin-logger-xz47w -n kube-system | grep relative
time="2025-09-04T05:35:54Z" level=info msg="PodSandbox CGroups" absolutePath=/sys/fs/cgroup/kubelet/kubepods/besteffort/pod3e66220e-824c-4b8c-827b-2d7082ffedae/1335660bd4566352fdc7ac876dca861ab15bbffc2ae43dc9be7f6f71515aa801 relativePath=/kubelet/kubepods/besteffort/pod3e66220e-824c-4b8c-827b-2d7082ffedae/1335660bd4566352fdc7ac876dca861ab15bbffc2ae43dc9be7f6f71515aa801
time="2025-09-04T05:35:56Z" level=info msg="Container CGroups" absolutePath=/sys/fs/cgroup/kubelet/kubepods/besteffort/pod3e66220e-824c-4b8c-827b-2d7082ffedae/e694db8605a01de2f0e6980e51a4129d1f33766b8ff41fa9ee2f69a234736c63 relativePath=/kubelet/kubepods/besteffort/pod3e66220e-824c-4b8c-827b-2d7082ffedae/e694db8605a01de2f0e6980e51a4129d1f33766b8ff41fa9ee2f69a234736c63
core@ngopalak-kind:~/nri$ docker ps
CONTAINER ID   IMAGE                  COMMAND                  CREATED             STATUS             PORTS                       NAMES
7e2354e834ab   kindest/node:v1.34.0   "/usr/local/bin/entr…"   About an hour ago   Up About an hour                               kind-nri-cluster-worker
b238f8911a2d   kindest/node:v1.34.0   "/usr/local/bin/entr…"   About an hour ago   Up About an hour   127.0.0.1:42603->6443/tcp   kind-nri-cluster-control-plane
core@ngopalak-kind:~/nri$ docker exec -it 7e2354e834ab bash
root@kind-nri-cluster-worker:/# ls /sys/fs/cgroup/kubelet/kubepods/besteffort/pod3e66220e-824c-4b8c-827b-2d7082ffedae/1335660bd4566352fdc7ac876dca861ab15bbffc2ae43dc9be7f6f71515aa801
cgroup.controllers	cgroup.threads	 cpuset.cpus.effective	   hugetlb.1GB.rsvd.max      io.stat		  memory.oom.group     misc.events
cgroup.events		cgroup.type	 cpuset.cpus.partition	   hugetlb.2MB.current	     io.weight		  memory.peak	       misc.max
cgroup.freeze		cpu.idle	 cpuset.mems		   hugetlb.2MB.events	     memory.current	  memory.pressure      pids.current
cgroup.kill		cpu.max		 cpuset.mems.effective	   hugetlb.2MB.events.local  memory.events	  memory.reclaim       pids.events
cgroup.max.depth	cpu.max.burst	 hugetlb.1GB.current	   hugetlb.2MB.max	     memory.events.local  memory.stat	       pids.max
cgroup.max.descendants	cpu.pressure	 hugetlb.1GB.events	   hugetlb.2MB.numa_stat     memory.high	  memory.swap.current  pids.peak
cgroup.pressure		cpu.stat	 hugetlb.1GB.events.local  hugetlb.2MB.rsvd.current  memory.low		  memory.swap.events   rdma.current
cgroup.procs		cpu.weight	 hugetlb.1GB.max	   hugetlb.2MB.rsvd.max      memory.max		  memory.swap.high     rdma.max
cgroup.stat		cpu.weight.nice  hugetlb.1GB.numa_stat	   io.max		     memory.min		  memory.swap.max
cgroup.subtree_control	cpuset.cpus	 hugetlb.1GB.rsvd.current  io.pressure		     memory.numa_stat	  misc.current
root@kind-nri-cluster-worker:/# crictl info | grep -i cgroup
          "cgroupWritable": false,
            "ShimCgroup": "",
            "SystemdCgroup": false

ngopalak-redhat avatar Sep 01 '25 09:09 ngopalak-redhat

@ngopalak-redhat @askervin This looks a bit off. Are you sure this really works ?

I think the resolution algorithm should be different depending on whether the runtime is using the systemd or the cgroupfs driver. And since NRI does not tell the plugin this at the moment, I think you probably need to either try to decide based on the syntax (which I'm not sure is always possible) or probe (resolve both and see which one exists).

If we decide to provide such a helper, it should not be part of pkg/api. I think it should go to pkg/plugin.

klihub avatar Sep 01 '25 11:09 klihub

@ngopalak-redhat @askervin This looks a bit off. Are you sure this really works ?

I think the resolution algorithm should be different depending on whether the runtime is using the systemd or the cgroupfs driver. And since NRI does not tell the plugin this at the moment, I think you probably need to either try to decide based on the syntax (which I'm not sure is always possible) or probe (resolve both and see which one exists).

If we decide to provide such a helper, it should not be part of pkg/api. I think it should go to pkg/plugin.

Thanks @klihub for the review. Happy to move to pkg/plugin.

For the first point, systemd and cgroupfs are two different ways to manage the same cgroups v2. Debugging further, I see kubepods-burstable-pod123.slice:cri-containerd:container456 as a systemd path and kubepods-burstable-pod123.slice/cri-containerd-container456.scope as a cgroupfs path. The colons and slashes are different. I overlooked the colons. Will address this and resubmit the PR. Making it draft till then.

ngopalak-redhat avatar Sep 01 '25 13:09 ngopalak-redhat

@klihub I have re-opened the PR with the requested changes. Initially thought its a good-first-issue to pick up to get deep understanding of NRI plugins. But handling the path turned out to be more complex than I expected. Its a useful change for users needing access to CGroups directory.

ngopalak-redhat avatar Sep 02 '25 07:09 ngopalak-redhat

/cc @askervin PTAL

klihub avatar Sep 03 '25 06:09 klihub

If we decide to provide such a helper, it should not be part of pkg/api. I think it should go to pkg/plugin.

WDYT about this functionality as part of https://github.com/containerd/cgroups

akhilerm avatar Sep 03 '25 07:09 akhilerm

If we decide to provide such a helper, it should not be part of pkg/api. I think it should go to pkg/plugin.

WDYT about this functionality as part of https://github.com/containerd/cgroups

@akhilerm Thanks a lot for your inputs/suggestion. The repo cgroups is used by containerd runtime itself. The plugins run as daemonsets and invoked via ttRPC over a unix socket nri.sock. The cgroups relative path is sent over the api call from containerd/cri-o to NRI plugins. Hence the absolute path function added in this PR uses the elements in the relative path and the node's /proc/mount to find out the cgroups V2 path. I feel it's better suited to remain local to the nri repository to avoid adding complexity to the core cgroups library

ngopalak-redhat avatar Sep 03 '25 10:09 ngopalak-redhat

Yeah I was wondering as well if there's either existing utilities in https://github.com/opencontainers/cgroups or https://github.com/containerd/cgroups that could be used (or added) and could be used for the implementation.

also cc @kolyshkin - who may have a better picture on what's already available (or missing).

thaJeztah avatar Sep 03 '25 11:09 thaJeztah

@ngopalak-redhat, thanks for picking up this task! This will definitely be helpful, and as you already saw, it's really non-trivial and easy to get wrong.

I tried it out with cri-o 1.32 systemd, containerd 1.17.27 systemd, and containerd 2.1beta cgroupfs. First two cases were fine, but in the last paths were interpreted absolute as is even if they were not:

[root@fedora-42 ~]#  ./logger -events createcontainer,runpodsandbox -enable-cgroups-log -idx 10 2>&1 | grep --color=auto -i cgroup &
[1] 8819
[root@fedora-42 ~]# kubectl create -f pod0.yaml
pod/pod0 created
[root@fedora-42 ~]# time="2025-09-03T14:44:54Z" level=info msg="RunPodSandbox:      cgroup_parent: /kubepods/besteffort/pod587e7105-0ad9-4c3c-9439-adb67738d658"
time="2025-09-03T14:44:54Z" level=info msg="RunPodSandbox:      cgroups_path: /kubepods/besteffort/pod587e7105-0ad9-4c3c-9439-adb67738d658/c816898460502f26511528dd406d00bed8397da49095f137069e7b1fefec7d57"
time="2025-09-03T14:44:54Z" level=info msg="PodSandbox CGroups" absolutePath=/kubepods/besteffort/pod587e7105-0ad9-4c3c-9439-adb67738d658/c816898460502f26511528dd406d00bed8397da49095f137069e7b1fefec7d57 relativePath=/kubepods/besteffort/pod587e7105-0ad9-4c3c-9439-adb67738d658/c816898460502f26511528dd406d00bed8397da49095f137069e7b1fefec7d57

I have fixed the usecase with cgroupfs and containerd and updated the description.

ngopalak-redhat avatar Sep 04 '25 07:09 ngopalak-redhat

Yeah I was wondering as well if there's either existing utilities in https://github.com/opencontainers/cgroups or https://github.com/containerd/cgroups that could be used (or added) and could be used for the implementation.

also cc @kolyshkin - who may have a better picture on what's already available (or missing).

@thaJeztah @ngopalak-redhat @askervin A possible (and I'd say also preferable) alternative to this would be to expose the corresponding path construction/resolution functions from either (or both) of the above packages for public consumption at a suitable pkg-subpath. Then, if we still wanted to provide a NRI plugin helper wrapper for it here, we'd import and use it here. Or alternatively we could do the same on the runtime side (where there is more explicit knowledge about the cgroup driver configuration) and pass it to NRI plugins as a new ResolvedCgroupPath container property.

klihub avatar Sep 04 '25 08:09 klihub