Duplicate kube_horizontalpodautoscaler_spec_target_metric

[dmitriishaburov]

What happened:

Duplicate kube_horizontalpodautoscaler_spec_target_metric causing issues with Prometheus 2.52.0+ Due to new duplicates detection mechanism, I'm seeing following errors in Prometheus:

ts=2024-05-28T07:15:26.730Z caller=scrape.go:1738 level=warn component="scrape manager" scrape_pool=serviceMonitor/prometheus/kube-prometheus-stack-kube-state-metrics/0 target=http://172.17.60.213:8080/metrics msg="Error on ingesting samples with different value but same timestamp" num_dropped=12

After checking kube-state metrics I've found that metrics for HPA are duplicated:

cat metrics | grep 'kube_horizontalpodautoscaler_spec_target_metric{namespace="dummy",horizontalpodautoscaler="dummy",metric_name="cpu",metric_target_type="utilization"}'
kube_horizontalpodautoscaler_spec_target_metric{namespace="dummy",horizontalpodautoscaler="dummy",metric_name="cpu",metric_target_type="utilization"} 100
kube_horizontalpodautoscaler_spec_target_metric{namespace="dummy",horizontalpodautoscaler="dummy",metric_name="cpu",metric_target_type="utilization"} 100
kube_horizontalpodautoscaler_spec_target_metric{namespace="dummy",horizontalpodautoscaler="dummy",metric_name="cpu",metric_target_type="utilization"} 100

HPA itself have 3 separate entries for CPU, but there's different on container:

k get hpa dummy -oyaml
  metrics:
  - containerResource:
      container: dummy-0
      name: cpu
      target:
        averageUtilization: 100
        type: Utilization
    type: ContainerResource
  - containerResource:
      container: dummy-1
      name: cpu
      target:
        averageUtilization: 100
        type: Utilization
    type: ContainerResource
  - containerResource:
      container: dummy-2
      name: cpu
      target:
        averageUtilization: 100
        type: Utilization
    type: ContainerResource

What you expected to happen: kube-state-metrics not producing duplicate metrics, probably by adding container label (?)

How to reproduce it (as minimally and precisely as possible):

• Create HPA as posted above
• Use Prometheus 2.52.0+ as scraper

Environment:

• kube-state-metrics version: v2.12.0
• Kubernetes version (use kubectl version): v1.28.9-eks-036c24b
• Cloud provider or hardware configuration: EKS