icinga2 icon indicating copy to clipboard operation
icinga2 copied to clipboard

Incorrect interpretation of units in perfdata normalization

Open gino0631 opened this issue 3 years ago • 0 comments

Describe the bug

As documented, Icinga normalizes performance data before passing it to metric backends, so that MB, MiB and similar become bytes. MB is interpreted as base 10, so that if a plugin reports 33762MB, 33762000000 bytes are recorded.

The problem is that many plugins historically calculate MB and similar using base 2 (actually, I'm not aware of a plugin that would use base 10). For instance, running /usr/lib64/nagios/plugins/check_disk --help will output the following explanation:

Note: kB/MB/GB/TB are still calculated as their respective binary units due to backward compatibility issues.

The change of behaviour in Icinga was most probably introduced with the support for new UoMs, as noticed in #9060.

To Reproduce

  1. Configure disk monitoring using disk command (this is by default for the monitoring host).
  2. Run the relevant plugin /usr/lib64/nagios/plugins/check_disk -c 10% -w 20% -m manually to see the results.
  3. Run /usr/lib64/nagios/plugins/check_disk -c 10% -w 20% -m -u MiB and verify that the numbers are the same, though the units are shown as MiB now.
  4. Check the performance data recorded in the InfluxDB. It will be normalized by multiplying by 1000*1000.

Expected behavior

Binary MB should be normalized by multiplying by 1024*1024, so that 33762MB should become 35402022912 bytes.

Your Environment

  • Version used (icinga2 --version): r2.13.3-1
  • Operating System and version: Linux CentOS 7 (Core)
  • Enabled features (icinga2 feature list): api checker ido-pgsql influxdb mainlog notification
  • Version of check_disk (/usr/lib64/nagios/plugins/check_disk --version): v2.3.3 (nagios-plugins 2.3.3)

Possible workarounds

By default, check_disk is executed with -m option, so that the units are MB. It is possible to put vars.disk_units = "MiB" variable for disk command, so that the units become MiB, and get translated correctly.

However, not all plugins support such options. For example, Manubulon SNMP plugin check_snmp_storage.pl uses binary MB units (it can be verified by checking the source code), and there are no options to change that, so the fix is to modify the source code to output 'MiB' instead of 'MB'.

Another case is /usr/lib64/nagios/plugins/check_mem.pl which outputs KBs which actually mean KiBs.

I'm not sure if the right way is to change all the plugins to use the new unit system though. Maybe there should be an option in Icinga how to interpret MBs and similar - using base 10 or base 2.

gino0631 avatar Aug 15 '22 11:08 gino0631