PromQL: add an annotation (warning) for too many resets in a counter

[bboreham]

Proposal

Users frequently come with questions about absurd results from rate, etc., when their problem is that the underlying data is oscillating.

Here is an example of oscillating underlying data:

We could warn when this is observed inside PromQL functions that take counters. For instance, if the query is rate(foo[5m]), and we see 5 data points with 4 resets, that would be "too many". Some number of resets is expected; a moderate number should not be warned about.

Suppose we set the bar at resets / points > 0.6 ?

[a3ylf]

Can this proposal be implemented already? If so, I would like to work on it.

[roidelapluie]

0.6 with the second condition that there should be at least 4 samples.

[bboreham]

I don't mind your suggestion, but doing the math:

1 sample: can't have any resets. 2 samples: can have 1 reset or 0.5 ratio. 3 samples: can have 2 resets or 0.66 ratio, if all three points are descending.

What's the reason not to warn on the 3rd case?

[beorn7]

Hello from the bug scrub!

An open PR is under review for this, leaving open.