Cluster Disk Usage is broken since there is no nodename label

[oryagel]

In K8s Cluster dashboard the disk usage panel uses node_filesystem_size{nodename=~"$node"}. In my Prometheus instance (and I guess that in other as well), this metric doesn't have a nodename label. What am I missing here?

[ghost]

Same issue

[vpistis]

same issue

[duanyifei1937]

same issue

[WesleyCharlesBlake]

Same here.

Any suggested workarounds?

[vpistis]

For Cluster Disk Usage gauge I changed the query to:

sum (container_fs_usage_bytes{device=~"^/dev/[nsx][v][bdm][a-z].+$",id="/",instance=~"^$node"}) / sum (container_fs_limit_bytes{device=~"^/dev/[nsx][v][bdm][a-z].+$",id="/",instance=~"^$node"})

and for me seems ok.

I changed also the queries for the Cluster Disk Capacity: usage:

sum (container_fs_usage_bytes{device=~"^/dev/[xv][sv]d[a-z][1-9]$",id="/",instance=~"$node"})

limit:

sum (container_fs_limit_bytes{device=~"^/dev/[xv][sv]d[a-z][1-9]$",id="/",instance=~"$node"})

*I use kubernetes 1.9.6

[WesleyCharlesBlake]

I tried a bunch of different queries and got results, but this seems more accurate. Thanks @vpistis . I will play around with the queries some more and let you know how I go. Cheers

[ghost]

I'm using this config for Prometheus, which contains nodename fix. The only difference from default config is that

          - source_labels: [__meta_kubernetes_pod_node_name]
            action: replace
            target_label: nodename

added to kubernetes-service-endpoints. I'm using Prometheus nodeExporter v0.15.2..

You can find more details here https://scalablesystem.design/categories/ds101/

[WesleyCharlesBlake]

awesome thanks @artyomboyko ,

Including this:

          - source_labels: [__meta_kubernetes_pod_node_name]
            action: replace
            target_label: nodename

To the prometheus kubernetes-service-endpoints config works like a charm as well. Thanks for this!

[goooseman]

I had the same problem (no disk usage info in Grafana), but in my case it was version of NodeExporter. I was using 0.17, but in 0.16 they've changed node_filesystem_size -> node_filesystem_size_bytes. Rolled back to 0.15.2 and now it works.

[seboudry]

If you're using a newer version of node-exporter (version 0.16 or 0.17), you can use job rules defines on upgrade guide: https://github.com/prometheus/node_exporter/blob/master/docs/V0_16_UPGRADE_GUIDE.md#use-recording-rules This way older and newer metrics renamed both exists (but it take more space on prometheus).