helm-charts
helm-charts copied to clipboard
grafana
[grafana] Add provisioning of new unified alerting available in version 9.1
Unified alerting provisioning was added in grafana version 9.1.
https://grafana.com/docs/grafana/latest/administration/provisioning/#alerting
Please add possibility to provision with helm chart following resources:
- Rules
- Contact points
- Notification policies
This has become a big headache for us as we introduce new Alerts in our provisioned dashboards. We've been generating these dashboards for some time now using IaC approaches where the dashboard templating is versioned in git. In the legacy system, introducing a new Alert was as simple as introducing a new rule in Panel code, deploying it, and allowing the provisioned dashboards sidecar to pick up and reflect those changes reliably.
After adopting Unified Alerts, we've noticed that these updates do not trigger subsequent migration passes over dashboards with introduced changes and therefore no new Alerts are being reflected after being deployed. We've found ourselves conducting force rollbacks of the Grafana DB to move temporarily back to Legacy Alerts, then re-migrate with the new Alert discovered. At that point we are back on the Unified Alert system with a rule defined in an IaC paradigm, but the journey is complex and also requires rebuilding notification templates, not to mention this migration scheme will probably be deprecated at some point.
It looks like this new provisioning support can solve this problem. Getting first-class support for this feature in Helm would be a huge quality of life improvement.
In addition to #1720 which adds support for adding alerts by customising the chart values.yaml, would it make sense to also include an approach similar to the configmap sidecar listener that is being used for dashboards, datasources and notifiers?
Is there a method for loading envFromSecret into the alerting sidecar? For things like slack tokens and whatnot
Is it correctly understood that #1720 handles rules and contact points, but that notification policies still cannot be configured via the Helm charts? If so, is that on the roadmap?
Is there a method for loading envFromSecret into the alerting sidecar? For things like slack tokens and whatnot
The sidecar approach already enables provisioning contact points from secrets, so that can be used for slack tokens.
Is it correctly understood that #1720 handles rules and contact points, but that notification policies still cannot be configured via the Helm charts? If so, is that on the roadmap?
I am not sure if notification policies can be provisioned through values.yaml, but they can be provisioned in the same alerting configuration file when using the alerting sidecars (setting sidecar.alerts.enabled to true and providing a configMap/secret with the configuration from the docs).
