node_exporter
node_exporter copied to clipboard
prometheus
Even though node_intr_total metrics is of counter type, it's value reduced
Host operating system: output of uname -a
Linux csd01lab-ddeio-0 3.10.0-1160.15.2.el7.x86_64 #1 SMP Thu Jan 21 16:15:07 EST 2021 x86_64 x86_64 x86_64 GNU/Linux
node_exporter version: output of node_exporter --version
node_exporter, version 1.0.1
Are you running node_exporter in Docker?
No. node_exporter is run on VNF based on openstack
What did you do that produced an error?
No specific scenario.
What did you expect to see?
node_intr_total metrics value should have same value as that of last scrape interval OR increase.
What did you see instead?
node_intr_total metrics value reduced compared to previous value.
Is it a known issue? If yes, is fix availble in recent release?
The exporter only reports what the kernel tells it to. So this is likely a bug in your kernel version.
This sounds like https://lore.kernel.org/lkml/[email protected]/
Looking at the supplied plot, it appears to have decreased by 2^32.
Good find. I'm not sure how easy it will be to actually work around that problem, other than exposing the individual per-cpu counters directly so resets can be handled.
Is this a problem? You'd rate() over this which should handle the reset, right?
Is this a problem? You'd rate() over this which should handle the reset, right?
It's not quite the same as a typical counter reset, since this counter is comprised of multiple individual counters tracked by the kernel. When just one of those individual kernel 32-bit counters rolls over, it causes node_intr_total metrics to go backwards. But it hasn't wrapped around from zero, so applying a rate() or increase() to that would not be mathematically correct.