[prometheusexporter] Expose native histograms

[krisztianfekete]

Component(s)

exporter/prometheus

What happened?

Description

Currently, prometheusreceiver supports native histograms, but prometheusexporter cannot expose them (because Prometheus cannot expose native histograms in text format yet).

Steps to Reproduce

1. Instrument your code with native histograms
2. Scrape the metrics with prometheusreceiver by following the instructions here: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/f5d54da37ae664e1a0b6934a61d38f1afab5501f/receiver/prometheusreceiver/README.md#prometheus-native-histograms

Expected Result

prometheusexporter should expose native histograms

Actual Result

Logs via debugexporter:

{"level":"error","ts":1718965637.50062,"caller":"[email protected]/accumulator.go:94","msg":"failed to translate metric","kind":"exporter","data_type":"metrics","name":"prometheus","data_type":"\u0004","metric_name":"gloo_mesh_translation_time_sec","stacktrace":"github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*lastValueAccumulator).addMetric
\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/accumulator.go:94
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*lastValueAccumulator).Accumulate
\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/accumulator.go:71
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*collector).processMetrics
\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/collector.go:88
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*prometheusExporter).ConsumeMetrics
\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/prometheus.go:85
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsRequest).Export
\tgo.opentelemetry.io/collector/[email protected]/exporterhelper/metrics.go:59
go.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send
\tgo.opentelemetry.io/collector/[email protected]/exporterhelper/timeout_sender.go:49
go.opentelemetry.io/collector/exporter/exporterhelper.(*baseRequestSender).send
\tgo.opentelemetry.io/collector/[email protected]/exporterhelper/common.go:37
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
\tgo.opentelemetry.io/collector/[email protected]/exporterhelper/metrics.go:158
--

Collector version

v0.103.0

Environment information

N/A

OpenTelemetry Collector configuration

receivers:
  prometheus:
    config:
      global:
        evaluation_interval: 1m
        scrape_interval: 15s
        scrape_timeout: 15s
        scrape_protocols: [ PrometheusProto, OpenMetricsText1.0.0, OpenMetricsText0.0.1, PrometheusText0.0.4 ]
        scrape_configs:
          <your targets exposing native histograms>
          ...
pipelines:
  metrics/test:
    receivers:
    - prometheus
    exporters:
    - debug
    - prometheus

Log output

{"level":"error","ts":1718965637.50062,"caller":"[email protected]/accumulator.go:94","msg":"failed to translate metric","kind":"exporter","data_type":"metrics","name":"prometheus","data_type":"\u0004","metric_name":"gloo_mesh_translation_time_sec","stacktrace":"github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*lastValueAccumulator).addMetric
\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/accumulator.go:94
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*lastValueAccumulator).Accumulate
\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/accumulator.go:71
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*collector).processMetrics
\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/collector.go:88
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*prometheusExporter).ConsumeMetrics
\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/prometheus.go:85
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsRequest).Export
\tgo.opentelemetry.io/collector/[email protected]/exporterhelper/metrics.go:59
go.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send
\tgo.opentelemetry.io/collector/[email protected]/exporterhelper/timeout_sender.go:49
go.opentelemetry.io/collector/exporter/exporterhelper.(*baseRequestSender).send
\tgo.opentelemetry.io/collector/[email protected]/exporterhelper/common.go:37
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
\tgo.opentelemetry.io/collector/[email protected]/exporterhelper/metrics.go:158
--

Additional context

No response

[github-actions[bot]]

Pinging code owners:

• exporter/prometheus: @Aneurysm9

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[wildum]

Hi, I tried it with the following config, and the native histograms are exposed just like they are exposed by my app:

receivers:
    prometheus:
      config:
        scrape_configs:
          - job_name: 'otel-collector'
            scrape_interval: 5s
            static_configs:
              - targets: ['localhost:9001']

exporters:
  prometheus:
    endpoint: "localhost:9010"
    const_labels:
      test_name: v1

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      exporters: [prometheus]
  telemetry:
    logs:
      level: debug

Native histogram metrics exposed by my app: golang_native_histogram_bucket{address="0.0.0.0",port="9001",le="+Inf"} 1523 golang_native_histogram_sum{address="0.0.0.0",port="9001"} 773369.1551833635 golang_native_histogram_count{address="0.0.0.0",port="9001"} 1523

Native histogram metrics exposed by the prometheus exporter: golang_native_histogram_bucket{address="0.0.0.0",instance="localhost:9001",job="otel-collector",port="9001",test_name="v1",le="+Inf"} 1284 golang_native_histogram_sum{address="0.0.0.0",instance="localhost:9001",job="otel-collector",port="9001",test_name="v1"} 656156.1718255163 golang_native_histogram_count{address="0.0.0.0",instance="localhost:9001",job="otel-collector",port="9001",test_name="v1"} 1284

Could you share your full setup?

[krisztianfekete]

Are you sure these are native histograms? They look like regular ones to me. The required options are seemingly missing from your receiver config: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/prometheusreceiver/README.md#prometheus-native-histograms, so my guess is that you're falling back to the legacy ones here.

EDIT: I added them to the original post, so it's easier to reproduce.

[wildum]

Ah you are right, I was missing the scape_protocols, I can reproduce it now, thanks :)

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/prometheus: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

Blocked by https://github.com/prometheus/client_golang/issues/1617.

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/prometheus: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

This is mostly unblocked after https://github.com/prometheus/client_golang/pull/1654. We just need to wait for a release. Exemplars are not yet supported

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/prometheus: @Aneurysm9 @dashpole @ArthurSens

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[ArthurSens]

Not stale, we're still waiting for a client_golang release

[GiedriusS]

v1.21.0 is out so this should be doable now 🚀

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/prometheusexporter: @Aneurysm9 @dashpole @ArthurSens

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[tharveybrown]

Is this being worked on now that it's unblocked?

[dashpole]

I believe it just needs someone to implement it.

[MoreraAlejandro]

Hey guys, if you don't mind I can work on this. I'll keep you posted!

[MoreraAlejandro]

I am working on this, I'll create a draft PR just to get quick feedback when possible. I'll continue updating the PR slow and steady since I am not that familiar with Prometheus and Native Histograms either.