helm-charts [kube-prometheus-stack] CPU Utilisation "No Data" due to bad recording rules

Describe the bug a clear and concise description of what the bug is.

The recording rule to create cluster:node_cpu:ratio_rate5m drops the cluster label from the resultant metric. This causes some grafana dashboard panels to report No Data as they're searching for cluster="$cluster'.

What's your helm version?

version.BuildInfo{Version:"v3.7.1", GitCommit:"1d11fcb5d3f3bf00dbe6fe31b8412839a96b3dc4", GitTreeState:"clean", GoVersion:"go1.16.9"}

What's your kubectl version?

v1.20.10

Which chart?

kube-prometheus-stack

What's the chart version?

36.0.1

What happened?

Dashboards are showing No Data for General/Kubernetes/Compute Resources/Cluster: CPU Utilisation.

What you expected to happen?

To see CPU Utilisation

How to reproduce it?

Install kube-prometheus-stack 36.0.1 with node-exporter (we're using the puppetdb exporter but the issue is at the recording rules, not the source) and grafana enabled.

Enter the changed values of values.yaml?

No response

Enter the command that you execute and failing/misfunctioning.

Not a command, but simple looking at the grafana dashboards

Anything else we need to know?

Pretty sure if the recording rule is changed to below it will work

Current
- rule: sum(rate(node_cpu_seconds_total{job="node-exporter",mode!="idle",mode!="iowait",mode!="steal"}[5m])) / count(sum by(cluster, instance, cpu) (node_cpu_seconds_total{job="node-exporter"}))
- result: {} 0.10346337677179257
Recommendation:
- rule: sum by (cluster) (rate(node_cpu_seconds_total{job="node-exporter",mode!="idle",mode!="iowait",mode!="steal"}[5m])) / count by (cluster) (sum by(cluster, instance, cpu) (node_cpu_seconds_total{job="node-exporter"}))
- result: {cluster="cluster"} 0.09313381330561242

Jun 22 '22 13:06 knack-roque

i think i'm experiencing something similar

nuked my kube-prometheus-stack and started from scratch just to be sure

v36.0.3 of the helm chart

nothing for cpu, memory, networking, or storage showing in the default dashboards

interestingly, the 'headlines' section at the top of 'compute resources' does show percentages.

otherwise, its mostly 'no data' in the panels.

Jun 23 '22 07:06 travnewmatic

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.