Give users the options to output native histograms, instead of classic histograms, from the metrics-generator

[09jvilla]

Is your feature request related to a problem? Please describe. The Tempo metrics-generator currently outputs a classic histogram, which creates the following 3 metrics:

traces_spanmetrics_latency_bucket
traces_spanmetrics_latency_count
traces_spanmetrics_latency_sum

Describe the solution you'd like Instead of outputting a classic histogram, I'd like to have the metrics-generator output a native histogram. I know that native histograms is still an experimental feature in Prometheus and Mimir, but its already showing a lot of promise. The benefits of native histograms are that they're much higher resolution than classical histograms, which, in this case, gives users a much more precise picture of the distribution of latencies of their spans. They do this at a fraction of the cost it would take to deliver that precision with a classical histogram.

Describe alternatives you've considered Just stick with outputting native histograms.

Additional context Metrics-generator should support outputting both classical and native histograms since not all users should be expected to have native histogram ingestion enabled on their Prometheus or Prometheus-compatible database.

[09jvilla]

cc @zhehao-grafana @krajorama for vis

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity. Please apply keepalive label to exempt this Issue.

[09jvilla]

Keep alive! Still valid request.

[09jvilla]

@zhehao-grafana mentioned he is following up with another user who was requesting this.

[mghildiy]

Maybe I can give it a try, if someone else is not.

[09jvilla]

@mghildiy - Thank you for volunteering! We would really appreciate your help on this. Give us another week or two to find a Tempo maintainer to review the PR you put together and provide input/guidance.

[09jvilla]

We'll post back once we find someone to pair you with.

[mghildiy]

Ok, I would meanwhile look into current histogram implementation.

[09jvilla]

Sounds great!

[krajorama]

I've made a short POC that maybe of use: https://github.com/grafana/tempo/pull/3288 , untested

[zalegrala]

Tempo now has the ability to configure which kind of histograms to generate on a per-tenant basis. I believe we can close this, but feel free to reopen if we need some follow up.

[09jvilla]

This is awesome, Zach! Congrats on getting this one done. :)

@zhehao-grafana and I will be reaching out this quarter to figure out how we can start testing this with customers who are already using the metrics-generator.