[BUG] koordlet failed to collect CPU/memory info from cgroup fs

[vivianzh]

What happened: koordlet failed to collect CPU/memory info from cgroup fs

What you expected to happen: Koordlet can collect info from cgroup

How to reproduce it (as minimally and precisely as possible):

1. Install koordinator: helm install koordinator koordinator-sh/koordinator --version 0.7;
2. Check koodlet logs.

Environment:

• App version: 0.7.0
• Kubernetes version (use kubectl version): Kubernetes v1.25.2
• Install details (e.g. helm install args): 0.7.0
• Containerd: containerd.io 1.6.8
• Kernel 5.10.18 + cgroup v1

[jasonliu747]

/area koordlet

[zwzhang0107]

cgroup v2 has not support yet, we will update this issue if this is updates, any comments is welcomed. https://github.com/koordinator-sh/koordinator/issues/407

[vivianzh]

More log updated: collect container kube-system/coredns-565d847f94-ksdgk/coredns cpu throttled failed, err open /host-cgroup/cpu/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3089d7a8_a983_4649_9088_39efe764e4b4.slice/cri-containerd-95f5f3fbc401447ebe9b25ba6be81be590995283f219e3ad469fb3bcc17a328b.scope/cpu.stat: no such file or directory. Then file /sys/fs/cgroup/cpu/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3089d7a8_a983_4649_9088_39efe764e4b4.slice/cpu.stat can ben opened and cgroup sys doesn't have cri-containerd-xxx directory.

[vivianzh]

The correct cgoup file in my system is below: [root@cpu]# find . -name 3089d7a8 ./system.slice/containerd.service/kubepods-burstable-pod3089d7a8_a983_4649_9088_39efe764e4b4.slice:cri-containerd:95f5f3fbc401447ebe9b25ba6be81be590995283f219e3ad469fb3bcc17a328b ./system.slice/containerd.service/kubepods-burstable-pod3089d7a8_a983_4649_9088_39efe764e4b4.slice:cri-containerd:2db2b26bb62261d56c6867e0f5734a1d0c6d609783add0e018627263b2ef2e68 ./kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3089d7a8_a983_4649_9088_39efe764e4b4.slice

[saintube]

The correct cgoup file in my system is below: [root@cpu]# find . -name 3089d7a8 ./system.slice/containerd.service/kubepods-burstable-pod3089d7a8_a983_4649_9088_39efe764e4b4.slice:cri-containerd:95f5f3fbc401447ebe9b25ba6be81be590995283f219e3ad469fb3bcc17a328b ./system.slice/containerd.service/kubepods-burstable-pod3089d7a8_a983_4649_9088_39efe764e4b4.slice:cri-containerd:2db2b26bb62261d56c6867e0f5734a1d0c6d609783add0e018627263b2ef2e68 ./kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3089d7a8_a983_4649_9088_39efe764e4b4.slice

@vivianzh could please show us the sub-directories of the pod cgroup?

e.g.

$ tree /sys/fs/cgroup/cpu/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3089d7a8_a983_4649_9088_39efe764e4b4.slice/

[vivianzh]

The correct cgoup file in my system is below: [root@cpu]# find . -name 3089d7a8 ./system.slice/containerd.service/kubepods-burstable-pod3089d7a8_a983_4649_9088_39efe764e4b4.slice:cri-containerd:95f5f3fbc401447ebe9b25ba6be81be590995283f219e3ad469fb3bcc17a328b ./system.slice/containerd.service/kubepods-burstable-pod3089d7a8_a983_4649_9088_39efe764e4b4.slice:cri-containerd:2db2b26bb62261d56c6867e0f5734a1d0c6d609783add0e018627263b2ef2e68 ./kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3089d7a8_a983_4649_9088_39efe764e4b4.slice

@vivianzh could please show us the sub-directories of the pod cgroup?

e.g.

$ tree /sys/fs/cgroup/cpu/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3089d7a8_a983_4649_9088_39efe764e4b4.slice/

[root@aee-icx ~]# tree /sys/fs/cgroup/cpu/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3089d7a8_a983_4649_9088_39efe764e4b4.slice/ /sys/fs/cgroup/cpu/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3089d7a8_a983_4649_9088_39efe764e4b4.slice/ ├── cgroup.clone_children ├── cgroup.procs ├── cpuacct.stat ├── cpuacct.usage ├── cpuacct.usage_all ├── cpuacct.usage_percpu ├── cpuacct.usage_percpu_sys ├── cpuacct.usage_percpu_user ├── cpuacct.usage_sys ├── cpuacct.usage_user ├── cpu.cfs_period_us ├── cpu.cfs_quota_us ├── cpu.core_group_cookie ├── cpu.core_tag ├── cpu.rt_period_us ├── cpu.rt_runtime_us ├── cpu.shares ├── cpu.stat ├── notify_on_release └── tasks

[saintube]

@vivianzh It is strange that no container cgroup exists under the pod cgroup path. Which kind of container runtime is in use? (runc, kata-runtime, etc.)

[vivianzh]

@vivianzh It is strange that no container cgroup exists under the pod cgroup path. Which kind of container runtime is in use? (runc, kata-runtime, etc.)

Runc

[zwzhang0107]

how about the version of operate system, ubuntu, centos?

[vivianzh]

how about the version of operate system, ubuntu, centos?

Centos8

[zwzhang0107]

emm.. this seems a little weird. the cgorup dir of container and pod are totally different. have you ever changed the config of kubelet, for example the --runtime-cgroups. it will be better if you can show the kubelet config and args.

[vivianzh]

emm.. this seems a little weird. the cgorup dir of container and pod are totally different. have you ever changed the config of kubelet, for example the --runtime-cgroups. it will be better if you can show the kubelet config and args.

I didn't add any cgroup related configs. /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=remote --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --pod-infra-container-image=registry.k8s.io/pause:3.8 #cat /var/lib/kubelet/config.yaml apiVersion: kubelet.config.k8s.io/v1beta1 authentication: anonymous: enabled: false webhook: cacheTTL: 0s enabled: true x509: clientCAFile: /etc/kubernetes/pki/ca.crt authorization: mode: Webhook webhook: cacheAuthorizedTTL: 0s cacheUnauthorizedTTL: 0s cgroupDriver: cgroupfs clusterDNS:

• 10.96.0.10 clusterDomain: cluster.local cpuManagerReconcilePeriod: 0s evictionPressureTransitionPeriod: 0s fileCheckFrequency: 0s healthzBindAddress: 127.0.0.1 healthzPort: 10248 httpCheckFrequency: 0s imageMinimumGCAge: 0s kind: KubeletConfiguration logging: flushFrequency: 0 options: json: infoBufferSize: "0" verbosity: 0 memorySwap: {} nodeStatusReportFrequency: 0s nodeStatusUpdateFrequency: 0s rotateCertificates: true runtimeRequestTimeout: 0s shutdownGracePeriod: 0s shutdownGracePeriodCriticalPods: 0s staticPodPath: /etc/kubernetes/manifests streamingConnectionIdleTimeout: 0s syncFrequency: 0s volumeStatsAggPeriod: 0s

[saintube]

@vivianzh Could you please show the container statuses of the pod (i.e. the status.containerStatuses field in pod yaml)? There may be no container-level cgroup for a non-running container.

[vivianzh]

kubectl get pods ffmpeg-convert-mp4-job1 -o yaml

containerStatuses:

• containerID: containerd://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx image: registry.cn-hangzhou.aliyuncs.com/eric-dev/ffmpeg:eric imageID: sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx lastState: {} name: ffmpeg ready: true restartCount: 0 started: true state: running: startedAt: "2022-10-11T21:43:02Z" hostIP:xxxxxxxxxxxx

[vivianzh]

#systemctl status containerd ├─kubepods-pod9f1d83b8_f9e8_4806_beaa_8457c4a8b286.slice:cri-containerd:6214a9ac3009e14874bb976fae5e18938bb8a48f6ecafc998b9750057d8d1d57 │ ├─9006 nginx: master process nginx -g daemon off; │ ├─9039 nginx: worker process │ └─9040 nginx: worker process └─kubepods-pod9f1d83b8_f9e8_4806_beaa_8457c4a8b286.slice:cri-containerd:c062ea3feca2ebecc7d0df87b7638ff909630b64031341eeaaf2969e51879bda └─8538 /pause It's containerd output, all cgroup files are writeen under system.slice/containerd.service, is it correct?

[vivianzh]

Any update for this issue?

[saintube]

@vivianzh Container-level cgroups in your node are under /system.slice/containerd.service, while koordlet supposes they should be under /kubepods.slice/kubepods-podxxxxx.slice/. It seems a misconfiguration problem of containerd cgroup driver. Please check if the cgroup driver configurations of both containerd and kubelet are correct.

[vivianzh]

@vivianzh Container-level cgroups in your node are under /system.slice/containerd.service, while koordlet supposes they should be under /kubepods.slice/kubepods-podxxxxx.slice/. It seems a misconfiguration problem of containerd cgroup driver. Please check if the cgroup driver configurations of both containerd and kubelet are correct. I have double checked cgroup driver configurations, no issues are found. This is my output of "systemctl statu containerd": ● containerd.service - containerd container runtime Loaded: loaded (/etc/systemd/system/containerd.service; disabled; vendor preset: disabled) Drop-In: /etc/systemd/system/containerd.service.d └─proxy.conf Active: active (running) Docs: https://containerd.io Main PID: 23097 (containerd) Tasks: 22 Memory: 118.0M CGroup: /system.slice/containerd.service

Is the output "CGroup: /system.slice/containerd.service" correct here? Could you share your correct output of containerd and i can find more clues.

[saintube]

@vivianzh Container-level cgroups in your node are under /system.slice/containerd.service, while koordlet supposes they should be under /kubepods.slice/kubepods-podxxxxx.slice/. It seems a misconfiguration problem of containerd cgroup driver. Please check if the cgroup driver configurations of both containerd and kubelet are correct. I have double checked cgroup driver configurations, no issues are found. This is my output of "systemctl statu containerd": ● containerd.service - containerd container runtime Loaded: loaded (/etc/systemd/system/containerd.service; disabled; vendor preset: disabled) Drop-In: /etc/systemd/system/containerd.service.d └─proxy.conf Active: active (running) Docs: https://containerd.io Main PID: 23097 (containerd) Tasks: 22 Memory: 118.0M CGroup: /system.slice/containerd.service

Is the output "CGroup: /system.slice/containerd.service" correct here? Could you share your correct output of containerd and i can find more clues.

I think it is incorrect. I'm not sure if the kubelet also considers this pattern of cgroup paths abnormal.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. This bot triages issues and PRs according to the following rules: - After 90d of inactivity, lifecycle/stale is applied - After 30d of inactivity since lifecycle/stale was applied, the issue is closed You can: - Mark this issue or PR as fresh with /remove-lifecycle stale - Close this issue or PR with /close Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it has not had recent activity. This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, the issue is closed You can:
• Reopen this PR with /reopen Thank you for your contributions.