[kube-prometheus-stack] No "cluster" label, therefore wrong data in Grafana

[timur-enikeev]

Describe the bug a clear and concise description of what the bug is.

Grafana dashboards for Kubernetes use "cluster" label. But there is no "cluster" label in metrics.

E.g. there is "Memory Utilisation" panel in "Kubernetes / Compute Resources / Cluster" Grafana Dashboard. It has 1 - sum(:node_memory_MemAvailable_bytes:sum{cluster="$cluster"}) / sum(node_memory_MemTotal_bytes{job="node-exporter",cluster="$cluster"}) expression.

But node_memory_MemAvailable_bytes metric doesn't have "cluster" label. It is:

node_memory_MemAvailable_bytes{container="node-exporter", endpoint="http-metrics", instance="192.168.60.190:9100", job="node-exporter", namespace="monitoring", pod="kube-prometheus-stack-prometheus-node-exporter-rwnff", service="kube-prometheus-stack-prometheus-node-exporter"}

Therefore this label shows wrong value (if there are other Node Exporters that this Prometheus scratches).

Can it be because kube-proxy isn't available for my cluster? As I understand, I can't change the metricsBindAddress to 0.0.0.0 (becasue of my cloud provider restrictions).

What's your helm version?

version.BuildInfo{Version:"v3.5.2", GitCommit:"167aac70832d3a384f65f9745335e9fb40169dc2", GitTreeState:"dirty", GoVersion:"go1.15.7"}

What's your kubectl version?

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5", GitCommit:"c285e781331a3785a7f436042c65c5641ce8a9e9", GitTreeState:"clean", BuildDate:"2022-03-16T15:58:47Z", GoVersion:"go1.17.8", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.10", GitCommit:"a7a32748b5c60445c4c7ee904caf01b91f2dbb71", GitTreeState:"clean", BuildDate:"2022-02-16T11:18:16Z", GoVersion:"go1.16.14", Compiler:"gc", Platform:"linux/amd64"}

Which chart?

kube-prometheus-stack

What's the chart version?

35.0.3

What happened?

No response

What you expected to happen?

No response

How to reproduce it?

No response

Enter the changed values of values.yaml?

No response

Enter the command that you execute and failing/misfunctioning.

helm -n monitoring upgrade kube-prometheus-stack prometheus-community/kube-prometheus-stack --install -f values.yaml

Anything else we need to know?

No response

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[Morriz]

I hope somebody can shed light on this. Is "cluster" only available in Thanos setup perhaps? And as such these dashboards for non-Thanos installs are now broken?

[chrisduong]

I have another problem but relating to "cluster" label. My metrics comes from Thanos, so it has "cluster" label, however the current built-in dashboards in the chart, hide the "cluster" label in the dashboard view, and I don't know how to unhide it.

@Morriz could you help if you know the way, I had searched all the issues but couldn't find the answer. I'm just getting to know the Grafana dashboard.

[Morriz]

Our problem was related to mismatch of selectors if I remember correctly. Your problem seems like another issue.

[chrisduong]

Yes, and I've just figured out how to unhide the Variables for the builtin dashboards.

[chrisduong]

But it really looks those builtin dashboards in the Helm chart are for Thanos only.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[allenhaozi]

hi @timur-enikeev What about this problem? I had the same problem, and I found a solution: https://stackoverflow.com/questions/63852779/how-to-set-a-label-in-service-monitor-so-it-appears-in-prometheus-metrics

But this requirement must have a Service

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale[bot]]

This issue is being automatically closed due to inactivity.

[mars64]

Using kube-prometheus-stack v35.3.1 with Thanos disabled, I can confirm that a variable and reference of the form cluster="$cluster" doesn't appear to break if cluster is null/empty.

I stumbled on this issue thinking the same thing, but other dashboards worked properly. Upon closer inspection, my issue was that there were namespaces worth of workload that weren't running as some had believed, resulting in a bunch of N/A that prompted me to look for this.

That is, I wonder if this issue is no more than a red herring for a different misconfiguration.