pushgateway icon indicating copy to clipboard operation
pushgateway copied to clipboard

Add support for native histograms

Open beorn7 opened this issue 1 year ago • 6 comments

It's still an experimental feature, but it should be relatively easy to add support to the PGW.

beorn7 avatar Nov 29 '22 14:11 beorn7

For protobuf, that is.

To support pushing in the text format, first the text format has to support native histograms. Which will happen in OpenMetrics, not in the old text format, if at all. So then this depends on #400 .

beorn7 avatar Dec 06 '22 17:12 beorn7

@jan--f is working on this.

beorn7 avatar Nov 09 '23 15:11 beorn7

So far I think this will just work as is. So things to be done are:

  • [ ] add storage test for histograms
  • [ ] add api test for histograms
  • [ ] confirm that scraping in text format returns at least _sum and _count
  • [ ] improve UI to display histograms

The last item might benefit from aligning with improvements made in Prometheus itself?

jan--f avatar Nov 15 '23 14:11 jan--f

The Prometheus UI issue is https://github.com/prometheus/prometheus/issues/11269 . Currently, the UI just displays the Histogram.String() output. PGW can do the same for now. Note that the PGW UI uses its own bespoke query API to render the UI (GET http://pushgateway.example.org:9091/api/v1/metrics etc., see README.md). For now, it's probably fine to simply replace the value string, which is currently always a float in string form, with the string representation of the Histogram. For fancy graphical rendering, we needed to return the histogram in some structured form.

beorn7 avatar Nov 15 '23 15:11 beorn7

Displaying native histograms via Histogram.String() is in #596.

Otherwise pushgateway just works with native histograms.

Push via protobuf works as expected

2023-11-22-150837_891x784

Scraping via protobuf works as expected

2023-11-22-150924_877x446

Scraping with text format, give us an old-style histogram with a single bucket and a _count and _sum metric.

2023-11-22-151429_852x485

jan--f avatar Nov 22 '23 14:11 jan--f

Thank you very much for all the work so far.

The representation in the PGW UI is the "machine level" one, not the human-readable one that you can see in the Prometheus UI (also see my comment on https://github.com/prometheus/pushgateway/pull/596).

Another thought that crossed my mind: We do have an API that returns all the pushed metrics in JSON, see https://github.com/prometheus/pushgateway#url-2 . I wrongly assumed that the UI is using this API call, but it isn't (yet), it just uses Go HTML templating directly. However, that API endpoint also needs to support native histograms, because some users might use it for their own tooling. (And of course, a future UI might actually use the API endpoint as originally planned.) I assume some code is needed to create a makeshift JSON format.

beorn7 avatar Nov 23 '23 17:11 beorn7