`prometheus_scrape` input cannot parse metrics from `kube-state-metrics`

[MrDrMcCoy]

Bug Report

Describe the bug fluent-bit prometheus_scrape input cannot parse metrics from kube-state-metrics.

• Example log message if applicable:

[2022/07/08 18:49:17] [error] [input:prometheus_scrape:prometheus_scrape.0] error decoding Prometheus Text format
[2022/07/08 18:49:32] [error] [input:prometheus_scrape:prometheus_scrape.0] error decoding Prometheus Text format
[2022/07/08 18:49:47] [error] [input:prometheus_scrape:prometheus_scrape.0] error decoding Prometheus Text format
...

• Steps to reproduce the problem: Configure fluent-bit to scrape kube-state-metrics.

Expected behavior Metrics should be parsed correctly and sent to output.

Your Environment

• Version used:
  • Fluent-bit image: cr.fluentbit.io/fluent/fluent-bit:1.9.6
  • Kube-state-metrics helm chart version: 4.10.0
  • Kube-state-metrics image: registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.5.0
• Configuration:

[INPUT]
    tag kube_metrics.*
    name prometheus_scrape
    host kube-state-metrics.kube-state-metrics.svc.cluster.local
    port 8080
    metrics_path /metrics
    scrape_interval 15s

• Environment name and version (e.g. Kubernetes? What version?):
  • Kubernetes version: AKS 1.23.5
  • Fluent-bit helm chart version: 0.20.4

Additional context This has not worked for me since prometheus_scrape was introduced. Previously uploaded details for similar bug (#5541). @tarruda requested I open separate bug.

Sample output of metrics from our kube-state-metrics is here: kube-state-metrics.log Collected via running wget -qO- http://kube-state-metrics.kube-state-metrics.svc.cluster.local:8080/metrics in another container.

[tarruda]

I've sent a cmetrics pull request to https://github.com/calyptia/cmetrics/pull/127, just ran a test parsing your attached example and confirmed it works.

[MrDrMcCoy]

@tarruda any word on when this will get pulled into a Fluent Bit release?

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.