It would be good to support kubelet_volume_stats* metrics

[zysimplelife]

Hi

I would like to use this metrics from cadvisor to send alarm if any volumn uses more than 80% required storage in kubernetes environment. local-path-provisioner however doesn't support his metrics. do you have any plan to support it or any reason to not support it?

there is a similar question on forum https://forums.rancher.com/t/kubelet-volume-stats-not-in-prometheus-stats/12390/2

[yasker]

We don't have an idea about it since we just haven't got time to look into it.

Also, PRs are welcome.

[just1900]

Also running into this issue, after digging into it, I found that kubelet will expose those kubelet_volume_stats_* metrics if we using local pv rather than hostpath pv, which means this issue can be solved after we fix this issue https://github.com/rancher/rancher/issues/22061.

Refs

For more detailed info, see the codelines below with kubelet v1.19.0, hostpath: https://github.com/kubernetes/kubernetes/blob/e19964183377d0ec2052d1f1fa930c4d7575bd50/pkg/volume/hostpath/host_path.go#L194-L198 vs local: https://github.com/kubernetes/kubernetes/blob/e19964183377d0ec2052d1f1fa930c4d7575bd50/pkg/volume/local/local.go#L129-L142

Try it out

Tested with manifests presented here, https://github.com/rancher/rancher/issues/22061#issuecomment-682110455

metrics can be found via kubelet's /metrics endpoint, but since localpath-provisioner do not support quota and just use a directory on the node for volume provisioning, so the related metrics' value is basically the same as what we could see via node-exporter

kubelet_volume_stats_available_bytes{namespace="default",persistentvolumeclaim="rke-localpv-pvc"} 2.3446458368e+10
# HELP kubelet_volume_stats_capacity_bytes [ALPHA] Capacity in bytes of the volume
# TYPE kubelet_volume_stats_capacity_bytes gauge
kubelet_volume_stats_capacity_bytes{namespace="default",persistentvolumeclaim="rke-localpv-pvc"} 4.2207490048e+10
# HELP kubelet_volume_stats_inodes [ALPHA] Maximum number of inodes in the volume
# TYPE kubelet_volume_stats_inodes gauge
kubelet_volume_stats_inodes{namespace="default",persistentvolumeclaim="rke-localpv-pvc"} 2.62144e+06
# HELP kubelet_volume_stats_inodes_free [ALPHA] Number of free inodes in the volume
# TYPE kubelet_volume_stats_inodes_free gauge
kubelet_volume_stats_inodes_free{namespace="default",persistentvolumeclaim="rke-localpv-pvc"} 2.430271e+06
# HELP kubelet_volume_stats_inodes_used [ALPHA] Number of used inodes in the volume
# TYPE kubelet_volume_stats_inodes_used gauge
kubelet_volume_stats_inodes_used{namespace="default",persistentvolumeclaim="rke-localpv-pvc"} 191169
# HELP kubelet_volume_stats_used_bytes [ALPHA] Number of used bytes in the volume
# TYPE kubelet_volume_stats_used_bytes gauge
kubelet_volume_stats_used_bytes{namespace="default",persistentvolumeclaim="rke-localpv-pvc"} 1.6994193408e+10

[flokli]

So, to make sure I understand this - local-path-provisioner should switch from hostpath pv to local pv, and then we'd get all these metrics out of the box?

[varac]

I am using local instead of hostpath with local-path-provisioner, and I have these metrics:

• kubelet_volume_stats_available_bytes
• kubelet_volume_stats_capacity_bytes
• kubelet_volume_stats_inodes
• kubelet_volume_stats_inodes_free
• kubelet_volume_stats_inodes_used
• kubelet_volume_stats_used_bytes

The problem is still that metrics for all volumes return the same numbers when they are on the same filesystem. So they are of very limited use. It would be great if at least kubelet_volume_stats_inodes_used and kubelet_volume_stats_used_bytes would return the actual disk usage of the host directory holding the local volume, instead of returning the disk usage stats from the filesystem.

[ugur99]

As discussed above, using local type volumes has some limitations and some of them are listed on sig-storage-local-static-provisioner best practices; on the other hand, currently, there is no such integration to get usage metrics of hostpath type volumes.

As a workaround, I wrote a simple project. It has a simple logic to generate capacity/usage metrics for pv's which use local-path storage class (actually any storage class using a static path for provisioning volumes on the nodes). For those who want to take a look: local-path-provisioner-pv-metrics-exporter