Stale alerts after upgrade to 9.x

[rda0]

What happened:

We have some alerts firing shown on /alerting/groups:

Expanding + See source leads to Rule could not be found.

A db query by rule_uid for the two uids shows:

+phd-influx:~# mysql --database=grafana -e "select annotations, dashboard_uid, panel_id from alert_rule where uid = 'dfvTe_Cnz';"
+------------------------------------------------------------------------------------------------------------------------+---------------+----------+
| annotations                                                                                                            | dashboard_uid | panel_id |
+------------------------------------------------------------------------------------------------------------------------+---------------+----------+
| {"__alertId__":"108","__dashboardUid__":"hui4eqjWz","__panelId__":"2","message":"PS-D filling level is out of range!"} | hui4eqjWz     |        2 |
+------------------------------------------------------------------------------------------------------------------------+---------------+----------+
+phd-influx:~# mysql --database=grafana -e "select annotations, dashboard_uid, panel_id from alert_rule where uid = 'ofvT6lCnz';"
+------------------------------------------------------------------------------------------------------------------------+---------------+----------+
| annotations                                                                                                            | dashboard_uid | panel_id |
+------------------------------------------------------------------------------------------------------------------------+---------------+----------+
| {"__alertId__":"109","__dashboardUid__":"hui4eqjWz","__panelId__":"4","message":"PS-E filling level is out of range!"} | hui4eqjWz     |        4 |
+------------------------------------------------------------------------------------------------------------------------+---------------+----------+

Note the dashboard_uid and panel_id. Looking at all the panels in the dashboard all show Alerts: 0. I suspect those are alerts in the old format, which were not correctly migrated or whatever.

Trying to find an alert created with the new system, reveals NULL in those fields:

+phd-influx:~# mysql --database=grafana -e "select annotations, dashboard_uid, panel_id from alert_rule where title = 'D_cryo_buffer';"
+-----------------------------------------------------------------+---------------+----------+
| annotations                                                     | dashboard_uid | panel_id |
+-----------------------------------------------------------------+---------------+----------+
| {"summary":"Alert for D-chamber buffer cryo pump temperature!"} | NULL          |     NULL |
+-----------------------------------------------------------------+---------------+----------+

How can we remove those stale alerts?

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Upgrade to v9.

Anything else we need to know?:

The general feedback from users is that migrated alerts are not very usable and all alerts need to be recreated using the new system.

Also it is not clear what the difference is between Alerting, Grafana Alerting, Legacy Alerting, [alerting] and [unified_alerting].

The configuration docs state:

[unified_alerting] enabled = ... The default value is false.

If [unified_alerting] == Grafana Alerting, aka new, then this is true for 9.x?

[alerting] enabled = Set to false to enable Grafana alerting and to disable legacy alerting engine. to disable Grafana alerting, set to true.

What is the default here?

Our grafana.ini uses the default for both (no explicit config), what does that mean?

Environment:

• Grafana version: 9.0.5
• Data source type & version: InfluxDB 1.8.10
• OS Grafana is installed on: Debian 10 Buster
• User OS & Browser:
• Grafana plugins:
• Others:

[grafanabot]

Please ask your question on community.grafana.com/. To avoid having your issue closed in the future, please read our CONTRIBUTING guidelines.

Happy graphing!