PV Capacity (kube_persistentvolume_capacity_bytes) #1134

[Gbotttt]

Hi Team,

I'm using Vmware environment and when i checked prometheus its not showing below metrics.

kubelet_volume_stats_inodes_used kubelet_volume_stats_used_bytes kubelet_volume_stats_available_bytes kubelet_volume_stats_capacity_bytes kubelet_volume_stats_inodes kubelet_volume_stats_inodes_free

I looked at the prom stack yaml and found only below jobs defined. Do you think i need to add some jobs related to kubelet ? if yes where and how shall i configure it. Also, if you can just give me basic idea as to how metrics are being pulled by prometheus. To what i understood it periodically fetches these metrics from prom stack yaml. correct me if i'm wrong.

Thanks

[Gbotttt]

PV Capacity (kube_persistentvolume_capacity_bytes) #1134--->this is the reference ticket.

[Gbotttt]

HI team, Can anyone look at this?

[dashpole]

@Gbotttt the metrics you list at the top are part of the metrics from the kubelet, and do not come from kube-state-metrics. /assign @dgrisonnet /triage accepted

[buger]

Generated by AI architect, can contain faulty statements

You're correct that the volume usage metrics you mentioned like kubelet_volume_stats_capacity_bytes come from the kubelet itself and not kube-state-metrics.

The kubelet exposes various stats about pods, containers, volumes etc. on an HTTP endpoint (typically :10255/metrics). Prometheus needs to be configured to scrape this kubelet endpoint in order to collect those metrics.

To enable kubelet metric scraping in Prometheus you need to:

1. Ensure the kubelet HTTP endpoint is exposed (should be by default)

2. Add a job configuration in Prometheus that scrapes the kubelet endpoints. For example:

  - job_name: 'kubelet'

    kubernetes_sd_configs:

    - role: node

    relabel_configs:

    - source_labels: [__address__]

      target_label: __address__

      replacement: kubernetes.default.svc:443

    scheme: https

    tls_config:

      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

3. Restart Prometheus to pick up the new job configuration.

Once Prometheus is scraping the kubelet metrics endpoint, you should start seeing all the kubelet_* metrics including the volume usage ones you mentioned.

The key is that kube-state-metrics only provides metrics derived from the Kubernetes API objects. The kubelet provides additional node/pod/container metrics via its own endpoints that need to be scraped independently.

Let me know if this helps explain why those volume metrics are missing! The main fix is getting Prometheus scraping the kubelet endpoint directly.

[dgrisonnet]

@Gbotttt what the relation between your initial request and https://github.com/kubernetes/kube-state-metrics/issues/1134?

To answer your initial questions:

I looked at the prom stack yaml and found only below jobs defined. Do you think i need to add some jobs related to kubelet ?

Yes

Also, if you can just give me basic idea as to how metrics are being pulled by prometheus. To what i understood it periodically fetches these metrics from prom stack yaml. correct me if i'm wrong.

I would advice you to look into https://prometheus.io/docs/prometheus/latest/getting_started/#configure-prometheus-to-monitor-the-sample-targets. Also if you want to make it simpler you can leverage prometheus-operator to configure your scrape targets: https://github.com/prometheus-operator/prometheus-operator.

The https://github.com/prometheus-operator/kube-prometheus/ project provides basic configuration to monitor a Kubernetes cluster with Prometheus and prometheus-operator which includes a configuration for kubelet: https://github.com/prometheus-operator/kube-prometheus/blob/main/manifests/kubernetesControlPlane-serviceMonitorKubelet.yaml

[k8s-triage-robot]

This issue has not been updated in over 1 year, and should be re-triaged.

You can:

• Confirm that this issue is still relevant with /triage accepted (org members only)
• Close this issue with /close

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted