Custom resource state metrics wildcard not working

[akselleirv]

What happened:

Metrics from custom resource is not produced when wildcard is used in kind field.

What you expected to happen:

The wildcard should produce metrics for all kinds.

How to reproduce it (as minimally and precisely as possible):

I have the following helm chart values file

rbac:
  extraRules:
    - apiGroups: ["azure.tnnova.io"]
      resources: ["*"]
      verbs: ["get", "list", "watch"]
    - apiGroups: ["apiextensions.k8s.io"]
      resources: ["customresourcedefinitions"]
      verbs: ["get", "list", "watch"]
customResourceState:
  enabled: true
  config:
    kind: CustomResourceStateMetrics
    spec:
      resources:
        - groupVersionKind:
            group: "azure.tnnova.io"
            version: "v1alpha1"
            kind: "UserAssignedIdentity"
          labelsFromPath:
            namespace: [metadata, namespace]
          metrics:
              - name: status_conditions
                help: "The condition of the resource"
                each:
                  type: StateSet
                  stateSet:
                    path: [status, conditions, "0"]
                    labelName: status
                    valueFrom: [status]
                    list: ["True", "False", "Unknown"]
                    labelsFromPath:
                      type: [type]

It produces the following metrics:

# HELP kube_customresource_status_conditions The condition of the resource
# TYPE kube_customresource_status_conditions stateset
kube_customresource_status_conditions{customresource_group="azure.tnnova.io",customresource_kind="UserAssignedIdentity",customresource_version="v1alpha1",namespace="s01234-example-system",status="False",type="Ready"} 0
kube_customresource_status_conditions{customresource_group="azure.tnnova.io",customresource_kind="UserAssignedIdentity",customresource_version="v1alpha1",namespace="s01234-example-system",status="True",type="Ready"} 1
kube_customresource_status_conditions{customresource_group="azure.tnnova.io",customresource_kind="UserAssignedIdentity",customresource_version="v1alpha1",namespace="s01234-example-system",status="Unknown",type="Ready"} 0

However, when I replace kind="UserAssignedIdentity" to kind="*" , like:

...
        - groupVersionKind:
            group: "azure.tnnova.io"
            version: "v1alpha1"
            kind: "*"
...

Then it no longer produces any metrics. I have verified that the kube-state-metrics service account has list,watch,get on customresourcedefintions.

Anything else we need to know?:

Environment:

• kube-state-metrics version: 2.12.0
• Kubernetes version (use kubectl version): v1.30.0
• Cloud provider or hardware configuration: kind
• Other info:

[logicalhan]

/triage accepted /assign @rexagod

[Thib17]

I was able to reproduce the issue inv2.12.0 but not in v2.11.0. After looking at the changes between those two versions I believe the bug is due to the following change in https://github.com/kubernetes/kube-state-metrics/compare/v2.11.0...v2.12.0#diff-60450a33adea08c953656dd1e78a80e9f3b279bbc7656dedf31fd1a0c7fc1196R119

				if header == lastHeader {
-					writer.stores[0].headers[i] = ""
-				} else {
-					lastHeader = header
+					writer.stores[0].headers = append(writer.stores[0].headers[:i], writer.stores[0].headers[i+1:]...)
+
+					// Do not increment the index, as the next header is now at the current index.
+					continue
				}

For some reason the behavior went:

• from [writer{stores: [store{headers:["h1"]}]}, writer{stores: [store{headers:["h1"]}]}] => [writer{stores: [store{headers:["h1"]}]}, writer{stores: [store{headers:[""]}]}]
• to___ [writer{stores: [store{headers:["h1"]}]}, writer{stores: [store{headers:["h1"]}]}] => [writer{stores: [store{headers:["h1"]}]}, writer{stores: [store{headers:[]}]}]

Note that [""] is now [] It used to empty the string value of the headers before writing them to the output, but now it also change the number of headers. Therefore some writers are not present in the output anymore.

I would suggest to rollback to the previous behavior by going back to writer.stores[0].headers[i] = "" I tried this, and it appear to fix the bug, but it require to rewrite all the unitests of the function called SanitizeHeaders. The unitest are enforcing the removal of headers wich cause the bug afaik. What do you think ? I can prepare a PR to implement this fix.

[mrueg]

@rexagod can you take a look? I believe this was introduced in https://github.com/kubernetes/kube-state-metrics/pull/2270