CustomResource: no metrics if CRD apply after ksm starts

[ZhangsongLee]

What happened:

There is a issues when a CRD is not applied before starting ksm

• no HELP or TYPE will get exported

What you expected to happen:

• Not requiring a restart to expose the metrics

How to reproduce it (as minimally and precisely as possible):

• Started KSM with configuration for the CRD
• Sleep for 5min
• Applied a CRD
• Created a CR of the above CRD type
• The metrics don't get exposed
• Restart KSM
• Metrics will get exposed

Used Configuration:

custom-resource-state-config.yaml: |
  spec:
    resources:
    - groupVersionKind:
        group: test.demo.io
        version: "v1alpha1"
        kind: crdtest
      labelsFromPath:
        namespace: [metadata, namespace]
      metrics:
        - name: "custom_request_cpu"
          help: "request cpu metrics"
          each:
            type: Gauge
            gauge:
              path: [status, custom, containers]
              labelsFromPath:
                container_name: ["containerName"]
              valueFrom: [cpu]

Anything else we need to know?: I found a related issue https://github.com/kubernetes/kube-state-metrics/issues/2142, this issue is before starting ksm, CRD already existed.

Environment:

• kube-state-metrics commit: https://github.com/kubernetes/kube-state-metrics/commit/eac6d3b1c994bd7dbd76026188d2b374bf000f20
• Kubernetes version (use kubectl version): v1.26

[CatherineF-dev]

Could you try KSM v2.8.2?

There are some big changes around CRD after v2.8.2.

[ZhangsongLee]

But I used commit(https://github.com/kubernetes/kube-state-metrics/commit/eac6d3b1c994bd7dbd76026188d2b374bf000f20) after v2.8.2

[CatherineF-dev]

v2.8.2+ has some bugs, you can find them in issues.

I guess this might be related as well.

Is v2.8.2 working for you?

[ZhangsongLee]

Unfortunately, v2.8.2 is not working :(

• what is different from before is that#HELP #TYPEnow appears, but still no metrics

 # HELP kube_customresource_custom_request_cpu request cpu metrics
 # TYPE kube_customresource_custom_request_cpu gauge

[CatherineF-dev]

This feature is working in my cluster.

Could you try examples here https://github.com/kubernetes/kube-state-metrics/blob/main/docs/customresourcestate-metrics.md?

I guess user configuration is not correct, you can add logs and rebuild KSM to debug. https://github.com/kubernetes/kube-state-metrics/pull/2299

[dashpole]

/assign @CatherineF-dev /triage accepted

[CatherineF-dev]

Hi @ZhangsongLee, is this issue resolved or not?