kube-state-metrics no longer collects labels we need by default

[yuvipanda]

We use metrics (pods, nodes, etc) with their labels in many of our dashboards. v2.0 came out a few months ago, and it doesn't add any labels by default. So the dashboards here won't work on prometheus servers using kube-state-metrics >= 2.0.

To work around this, we need to either:

1. Stop depending on labels at all
2. Provide configuration for kube-state-metrics that collects these labels for the metrics

[minrk]

Arg! I guess the first step is to write down what labels we use. The chart appears to be here.

It looks like it's going to be prometheus.kubeStateMetrics.metricLabelsAllowlist here that we need to populate. I don't really understand the syntax of that option, though.

In the absence of that, pinning the prometheus chart to 13.* will use the older kube-state-metrics chart, since it appears to be bumped in 14.0.

[choldgraf]

If I understand @minrk correctly, the prometheus pin could fix this problem now? I'd be +1 on:

1. Pinning prometheus to 13.* to un-break this repository immediately
2. Opening an issue to track upgrading to prometheus 14.* as an enhancement, rather than fixing a bug

[minrk]

I just ran into this and had occasion to figure it out. Deployments need the config:

prometheus:
   kube-state-metrics:
     metricLabelsAllowlist:
       - pods=[app,component] # to select jupyterhub component pods
       - nodes=[cloud_google_com_gke_nodepool] # to group nodes by pool on GKE
       # - nodes[*] is probably fine for nodes, since they don't churn much, unlike pods