mtail
mtail copied to clipboard
Histogram lower bucket and "0" observations expectations
Hi,
thank you for mtail, I'm following up from #226 where it was reported mtail histogram buckets don't match Prometheus' specs/expectations, specifically with regard to 0 observations.
I can confirm that the issue is still present in rc50, IOW observing a 0 value increments _count but not the lowest bucket:
$ cat issue-226.mtail
histogram foo_requests buckets -1, 2, 4, 6 by foo
/^(?P<foo>[a-z]+) (?P<time>\d+)$/ {
foo_requests[$foo] = $time
}
$ ./mtail -progs issue-226.mtail -logs /tmp/foo.log &
$ echo "c 0" >> /tmp/foo.log
$ curl -s localhost:3903/metrics | grep -i requests
# HELP foo_requests defined at issue-226.mtail:1:11-22
# TYPE foo_requests histogram
foo_requests_bucket{foo="c",prog="issue-226.mtail",le="2"} 0
foo_requests_bucket{foo="c",prog="issue-226.mtail",le="4"} 0
foo_requests_bucket{foo="c",prog="issue-226.mtail",le="6"} 0
foo_requests_bucket{foo="c",prog="issue-226.mtail",le="+Inf"} 0
foo_requests_sum{foo="c",prog="issue-226.mtail"} 0
foo_requests_count{foo="c",prog="issue-226.mtail"} 1
This caught us by surprise when using mtail's native histograms and exporting to Prometheus. The "add a -1 bucket" workaround does indeed work though I think histograms should match what users expect (i.e. all observations are <= their buckets). What do you think? Alternatively the issue could be documented (and perhaps linted for?)