Kubelet "cri_stats_provider" logs generated after CRI-O installation

[rodnymolina]

Let me start by saying that his is not a Sysbox issue, but I'm reporting it here because I'm expecting users to reproduce this behavior once they switch to CRI-O runtime as part of the Sysbox installation process in K8s clusters.

As part of the installation of the crio-deploy-k8s daemonset in the elected k8s nodes, kubelet will be restarted and the pre-existing pods will be re-instantiated. From that moment on, kubelet may start to dump logs like these ones every 10 seconds:

May 22 17:10:36 ip-172-20-42-62 kubelet[15230]: E0522 17:10:36.179214   15230 cri_stats_provider.go:376] Failed to get the info of the filesystem with mountpoint "/var/lib/containers/storage/overlay/d55f04c7ee1011670bae419c7d9d7a3a9d9fb620e01f7e61399a7950052ace5b/merged": failed to get device for dir "/var/lib/containers/storage/overlay/d55f04c7ee1011670bae419c7d9d7a3a9d9fb620e01f7e61399a7950052ace5b/merged": could not find device with major: 0, minor: 93 in cached partitions map.
May 22 17:10:36 ip-172-20-42-62 kubelet[15230]: E0522 17:10:36.188714   15230 cri_stats_provider.go:376] Failed to get the info of the filesystem with mountpoint "/var/lib/containers/storage/overlay/58197d684779f82964137f1bc50201322cfd8567cad0fd00e51f4dcaf11db28d/merged": failed to get device for dir "/var/lib/containers/storage/overlay/58197d684779f82964137f1bc50201322cfd8567cad0fd00e51f4dcaf11db28d/merged": could not find device with major: 0, minor: 88 in cached partitions map.
May 22 17:10:36 ip-172-20-42-62 kubelet[15230]: E0522 17:10:36.191654   15230 cri_stats_provider.go:376] Failed to get the info of the filesystem with mountpoint "/var/lib/containers/storage/overlay/3b6a438c4e5f0bbec943efd47f08773dcee925c0f32e08fcbfb2efa919f22462/merged": failed to get device for dir "/var/lib/containers/storage/overlay/3b6a438c4e5f0bbec943efd47f08773dcee925c0f32e08fcbfb2efa919f22462/merged": could not find device with major: 0, minor: 79 in cached partitions map.

Basically, kubelet's cadvisor/cri-stats component is complaining about being unable to reach overlayfs partitions associated to the pods running in this k8s node. The kubelet task being impacted (cri_stats_provider) is simply collecting system statistics, so this issue has no impact on the service.

Problem is being tracked here and here.