sanoid icon indicating copy to clipboard operation
sanoid copied to clipboard

fix: Warn when newest snapshot is in the future

Open koleo opened this issue 3 years ago • 0 comments

Description

Option --monitor-snapshots issues a Warning or a Critical when newest snapshot is too old based on the thresholds defined in the Sanoid configuration.

However, it reports an OK status when newest snapshot is in the future. This can happen when system time is wrong (in the future) for some time and then returns to normal. In such a situation, this may prevent the creation of new snapshots for a while and so we need to be alerted.

How to test

On a Linux Debian 10 server:

# Stop timesyncd
systemctl stop systemd-timesyncd.service

# Change date (+1 month)
date --set="$(date --date='next month')"

# take snapshots (they will be dated one month in the future)
./sanoid --take-snapshots

# Start timesyncd (the date will come back on time)
systemctl start systemd-timesyncd.service

# Monitor snapshots
## actual behaviour
./sanoid --monitor-snapshots ; echo $?
OK: all monitored datasets (tank/data2) have fresh snapshots
0
## new behaviour (this PR)
./sanoid --monitor-snapshots ; echo $?
WARN: tank/data2 newest daily snapshot is -29d -23h -56m -44s old (in the future), WARN: tank/data2 newest hourly snapshot is -29d -23h -56m -44s old (in the future), WARN: tank/data2 newest monthly snapshot is -4d -10h -29m -19s old (in the future)
1

Open questions

  • The decision to issue a Warning is arbitrary. It could also be a Critical.
  • We could emit a Warning when newest snapshot is in the near future, and a Critical when it's in the distant future, based on the same thresholds defined to detect recentness of the newest snapshots.
  • We could even add configuration options to define whether we want to issue a Warning, a Critical or just ignore such a situation. But that may be excessive for this infrequent event?

koleo avatar Jun 27 '21 12:06 koleo