Counter should start from 0

[mhoshino]

Current implementation is not following the below rule

https://prometheus.io/docs/practices/instrumentation/#avoid-missing-metrics

To avoid this, export 0 (or NaN, if 0 would be misleading) for any time series you know may exist in advance.

This is leading to situation's such as creating prometheus alert. For example

rate(my_metric[1m]) > 0

Will not fire alert when my_metric=1 but instead my_metric > 2. This is because my_metric does not start with 0 and start's with 1

[dkalashnik]

Usually that happen after fluentd restart. Maybe there would be a way to make some storage to persist metrics/values between reboots? Some sort of buffer on disk

[mhoshino]

Hi @dkalashnik . Do we really need to store this on buffer disk? The problem I am seeing is that, it needs to capture a message at least 2 times every time for it to generate alert after restart happens on fluentd. If it can start from 0 after the restart, I am assuming it will be able to fire the alert on the 1st message

[mescanef]

@mhoshino basicly its about https://github.com/prometheus/prometheus/issues/1673

[mhoshino]

It seems without this being resolved there counter for the prometheus ruby client does not start from 0

prometheus/client_ruby#141

[Kuckkuck]

+1 We still need this for error counting and doing delta alerting, otherwise the first counter will never be catched and you cannot use this metric for alerting.

[yeplaa]

@mhoshino @Kuckkuck have you found a solution to use fluentd-plugin-prometheus metrics from prometheus?