[receiver/prometheusreceiver] Can't scrape histogram and summary at the same time (Prometheus receiver)

[jjmm0312]

trafficstars

Hello. I want to collect histogram and summary at the same time. Prometheus exports the metric, but opentelemetry says error message.

Prometheus Metrics (I removed another labels like host, remote.) lettuce_command_completion_seconds{command="SET",quantile="0.5",} 0.0 lettuce_command_completion_seconds{command="SET",quantile="0.9",} 0.0

lettuce_command_completion_seconds_bucket{command="SET",le="0.001",} 7.0 lettuce_command_completion_seconds_bucket{command="SET",le="0.001048576",} 9.0

Opentelemetry collector ..."error": "BucketLabel or QuantileLabel is empty"...

Is it impossible to scrape histogram and summary at the same time?

I use opentelemetry-collector-contrib:0.56.0.

[github-actions[bot]]

Pinging code owners: @Aneurysm9 @dashpole. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

How are you producing those metrics? It looks like you are trying to have a single metric which is both a histogram and a summary, which isn't possible in prometheus. What # TYPE comment do you have on that metric?

[jjmm0312]

I use Spring boot library "micrometer-registry-prometheus" which exposes prometheus type metrics. The type is # TYPE lettuce_command_completion_seconds histogram

But the metric has quantile label or bucket label. The format looks like summary and histogram. I use default setting, so I'm not sure if it is impossible with opentelemetry to scrape both metrics.

[dashpole]

OpenTelemetry doesn't currently have a data format which is both a histogram and a summary, so i'm not sure we can correctly represent the histogram + summary "combo" metric you have.

There are a few things that we might be able to do:

1. Since your type comment asked for a histogram, and you provided all of the histogram series, we should be able to completely construct the histogram. We could emit a warning that we are dropping the summary metric points.
2. We could improve the error message to be less cryptic (strange that it is complaining about a missing bucket label, when you clearly have it)

Would you have found either of the above helpful?

You might be able to work around this by dropping the summary series using prometheus relabel rules (with something like):

  metric_relabel_configs:
  - source_labels: [__name__]
    regex: lettuce_command_completion_seconds
    action: drop

[jjmm0312]

I think first one is better for me. Because I can't collect all metrics because of the histogram metric. I'll use metric_relabel_configs until the feature is added. Thanks.

[dmitryax]

I'll use metric_relabel_configs until the feature is added.

I'm not sure we want to do anything to support this use case.

@Aneurysm9 @dashpole WDYT? Can we close the issue?

[dashpole]

I looked at the implementation, and it looks like it should have been sending the histogram-portion of the metric correctly already. The error described above just drops the summary point.

[dashpole]

I opened https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/13317 to improve the error message, but I don't think there is more we can do

[cmergenthaler]

Hey @dashpole, we have one service that also uses the described metric here. Although you are saying it should only drop the summary metric, it seems like the collector drops all metrics and prints a warn log with Failed to scrape Prometheus endpoint. Also the service is reported with 0 for the up metric.

[dashpole]

We are fixing this behavior in https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/24030

[cmergenthaler]

Awesome thanks!