mtail icon indicating copy to clipboard operation
mtail copied to clipboard

Histogram lower bucket and "0" observations expectations

Open filippog opened this issue 3 years ago • 0 comments

Hi, thank you for mtail, I'm following up from #226 where it was reported mtail histogram buckets don't match Prometheus' specs/expectations, specifically with regard to 0 observations.

I can confirm that the issue is still present in rc50, IOW observing a 0 value increments _count but not the lowest bucket:

$ cat issue-226.mtail 
histogram foo_requests buckets -1, 2, 4, 6 by foo

/^(?P<foo>[a-z]+) (?P<time>\d+)$/ {
  foo_requests[$foo] = $time
}
$ ./mtail -progs issue-226.mtail -logs /tmp/foo.log &
$ echo "c 0" >> /tmp/foo.log
$ curl -s localhost:3903/metrics | grep -i requests
# HELP foo_requests defined at issue-226.mtail:1:11-22
# TYPE foo_requests histogram
foo_requests_bucket{foo="c",prog="issue-226.mtail",le="2"} 0
foo_requests_bucket{foo="c",prog="issue-226.mtail",le="4"} 0
foo_requests_bucket{foo="c",prog="issue-226.mtail",le="6"} 0
foo_requests_bucket{foo="c",prog="issue-226.mtail",le="+Inf"} 0
foo_requests_sum{foo="c",prog="issue-226.mtail"} 0
foo_requests_count{foo="c",prog="issue-226.mtail"} 1

This caught us by surprise when using mtail's native histograms and exporting to Prometheus. The "add a -1 bucket" workaround does indeed work though I think histograms should match what users expect (i.e. all observations are <= their buckets). What do you think? Alternatively the issue could be documented (and perhaps linted for?)

filippog avatar Aug 10 '22 10:08 filippog