Monitoring rules for systemd units (monitoring_srv/prometheus/rules.yml)

[pirat013]

The current configuration for the systemd unit files are monitoring the active state like this:

• name: systemd-services-monitoring rules:
  • alert: service-down-pacemaker expr: node_systemd_unit_state{name="pacemaker.service", state="active"} == 0 labels: severity: page annotations: summary: Pacemaker service not running

This would lead into false positive report due to maintenance work or other task when the systemd units are stop by an admin. I would suggest to change the monitoring rule from active to failed:

• name: systemd-services-monitoring rules:
  • alert: service-failed-pacemaker expr: node_systemd_unit_state{name="pacemaker.service", state="failed"} == 1 labels: severity: page annotations: summary: Pacemaker service could not start or is crashed.

This would create less calls in regards to the situation a systemd unit is stop due to maintenance. If we would go this way we could think about to shorten the list and using a configuration like this:

• alert: HostSystemdServiceCrashed expr: node_systemd_unit_state{state="failed"} == 1 for: 1m labels: severity: page annotations: description: |- systemd service crashed VALUE = {{ $value }} LABELS = {{ $labels }} summary: Host systemd service crashed (instance {{ $labels.instance }})

[yeoldegrove]

@pirat013 I would not see anything that speaks against this change. Are you willing to submit a PR?

[pirat013]

@yeoldegrove sorry I didn't see your request. We may have to consider a combination of service is enabled and not started as well this would reflect the original idea better than my suggestion. I'll try to figure out this rule and can create a PR. But I can't say when it will happen.