grafana
grafana copied to clipboard
Stale alerts after upgrade to 9.x
What happened:
We have some alerts firing shown on /alerting/groups
:
Expanding + See source
leads to Rule could not be found.
A db query by rule_uid
for the two uids shows:
+phd-influx:~# mysql --database=grafana -e "select annotations, dashboard_uid, panel_id from alert_rule where uid = 'dfvTe_Cnz';"
+------------------------------------------------------------------------------------------------------------------------+---------------+----------+
| annotations | dashboard_uid | panel_id |
+------------------------------------------------------------------------------------------------------------------------+---------------+----------+
| {"__alertId__":"108","__dashboardUid__":"hui4eqjWz","__panelId__":"2","message":"PS-D filling level is out of range!"} | hui4eqjWz | 2 |
+------------------------------------------------------------------------------------------------------------------------+---------------+----------+
+phd-influx:~# mysql --database=grafana -e "select annotations, dashboard_uid, panel_id from alert_rule where uid = 'ofvT6lCnz';"
+------------------------------------------------------------------------------------------------------------------------+---------------+----------+
| annotations | dashboard_uid | panel_id |
+------------------------------------------------------------------------------------------------------------------------+---------------+----------+
| {"__alertId__":"109","__dashboardUid__":"hui4eqjWz","__panelId__":"4","message":"PS-E filling level is out of range!"} | hui4eqjWz | 4 |
+------------------------------------------------------------------------------------------------------------------------+---------------+----------+
Note the dashboard_uid
and panel_id
. Looking at all the panels in the dashboard all show Alerts: 0
. I suspect those are alerts in the old format, which were not correctly migrated or whatever.
Trying to find an alert created with the new system, reveals NULL
in those fields:
+phd-influx:~# mysql --database=grafana -e "select annotations, dashboard_uid, panel_id from alert_rule where title = 'D_cryo_buffer';"
+-----------------------------------------------------------------+---------------+----------+
| annotations | dashboard_uid | panel_id |
+-----------------------------------------------------------------+---------------+----------+
| {"summary":"Alert for D-chamber buffer cryo pump temperature!"} | NULL | NULL |
+-----------------------------------------------------------------+---------------+----------+
How can we remove those stale alerts?
What you expected to happen:
How to reproduce it (as minimally and precisely as possible):
Upgrade to v9.
Anything else we need to know?:
The general feedback from users is that migrated alerts are not very usable and all alerts need to be recreated using the new system.
Also it is not clear what the difference is between Alerting
, Grafana Alerting
, Legacy Alerting
, [alerting]
and [unified_alerting]
.
The configuration docs state:
[unified_alerting] enabled = ... The default value is
false
.
If [unified_alerting]
== Grafana Alerting
, aka new
, then this is true
for 9.x?
[alerting] enabled = Set to
false
to enable Grafana alerting and to disable legacy alerting engine. to disable Grafana alerting, set totrue
.
What is the default here?
Our grafana.ini
uses the default for both (no explicit config), what does that mean?
Environment:
- Grafana version: 9.0.5
- Data source type & version: InfluxDB 1.8.10
- OS Grafana is installed on: Debian 10 Buster
- User OS & Browser:
- Grafana plugins:
- Others:
Please ask your question on community.grafana.com/. To avoid having your issue closed in the future, please read our CONTRIBUTING guidelines.
Happy graphing!