prometheus-formula
prometheus-formula copied to clipboard
[BUG] Changes to alerts require reload of prometheus
Your setup
Formula commit hash / release tag
n/a
Versions reports (master & minion)
n/a
Pillar / config used
prometheus:
extra_files:
apache_rules:
file: service_rules/apache
component: alertmanager
config:
groups:
- name: 'apache.rules'
rules:
- alert: ApacheDown
expr: apache_up == 0
for: 0m
labels:
severity: critical
annotations:
summary: {% raw %} Apache down (instance {{ $labels.instance }}) {% endraw %}
description: {% raw %} "Apache down\n VALUE = {{ $value }}\n LABELS = {{ $labels }}" {% endraw %}
- alert: ApacheWorkersLoad
expr: (sum by (instance) (apache_workers{state="busy"}) / sum by (instance) (apache_scoreboard) ) * 100 > 80
for: 2m
labels:
severity: warning
annotations:
summary: {% raw %} Apache workers load (instance {{ $labels.instance }}) {% endraw %}
description: {% raw %} "Apache workers in busy state approach the max workers count 80% workers busy on {{ $labels.instance }}\n VALUE = {{ $value }}\n LABELS = {{ $labels }}" {% endraw %}
- alert: ApacheRestart
expr: apache_uptime_seconds_total / 60 < 1
for: 0m
labels:
severity: warning
annotations:
summary: {% raw %} Apache restart (instance {{ $labels.instance }}) {% endraw %}
description: {% raw %} "Apache has just been restarted.\n VALUE = {{ $value }}\n LABELS = {{ $labels }}" {% endraw %}
Bug details
Describe the bug
Changing the extra_files alerts appears to result in a restart of the alertmanager service but needs to restart the prometheus process too otherwise changes arent updated in prometheus.
Steps to reproduce the bug
- Highstate pillar with alerts.
- Remove an alert from the pillar
- Re highstate pillar
- Note alert has not been removed from prometheus
- Restart prometheus
- Note now alert has been removed
Expected behaviour
Prometheus service should be restarted on change to extra_files / alerts
Attempts to fix the bug
None yet.