rate() function giving spurious peak

[apeksharma]

What did you do? Plotting graph of a metric (say M) and the rate of that metric.

What did you expect to see? No deviation in normal graph of rate(M)

What did you see instead? Under which circumstances? Unexpected spike in the graph

Data points for that duration

Environment

• System information:

Don't have access to the system. Can get it if needed.

• Prometheus version:

  2.18.1

• Alertmanager version:

NA

• Prometheus configuration file: Don't have access to the system. Can get it if needed.

• Logs:

insert Prometheus and Alertmanager logs relevant to the issue here

[roidelapluie]

You probably have resets.

Can you please provide the raw metrics?

to do that, run mymetric{label="a"}[3h] in the "console" tabs.

[apeksharma]

Thanks Julien for responding.

PFA raw metrics. Also, adding graphs from the exact same duration for easy correlation. raw_metrics.txt

[roidelapluie]

You have a counter reset:

439 @1617442554.304 439 @1617442554.315 440 @1617442564.304 439 @1617442564.315 440 @1617442574.33 440 @1617442584.297

[roidelapluie]

It might be a bug in prometheus. Why is that scraped twice at the same time?

Can you are more about your configuration?

[apeksharma]

cc: @sanjeevpandey19: Sanjeev, would you be able to help with above questions?

[vsliouniaev]

Reminds me a bit of this issue: https://github.com/prometheus/prometheus/issues/6894

[roidelapluie]

I totally agree and I digged multiple times in the codebase without finding the bug yet...

[apeksharma]

I see. Can close this in favor of earlier one then. Thanks Julien and Vasily

[roidelapluie]

Can you still provide the requested info please?

[sanjeevpandey19]

Can you still provide the requested info please?

@roidelapluie what sort of specific information you need here.

Do you want to know if we have setup using operator or vanila Prometheus or you would need Prometheus.yaml itself ?

[roidelapluie]

I'd like the prometheus.yml to see the relabeling rules and the jobs, if possible.

[dhruvp-8]

@roidelapluie any update on this issue we are also facing the same issue in Prometheus 2.25.0 where in the rate() function has unexpected spikes in the data

[amey24kulkarni]

Team, Is there any update on this issue.. We're also facing similar issue. But with increase() function

[beorn7]

Hello from the bug scrub.

It's still mysterious what's happening here. The best we know is that the reason is #6894, so let's bundle activities there and close this one.