Add support for notifiers provisioning

[olivierlemasle]

This PR adds support for notifiers provisioning (= notification channels) https://grafana.com/docs/administration/provisioning/#alert-notification-channels

While notifiers is left empty, it does not modify anything. If notifier is not empty, it generates a Secret and mounts it as a volume, like for datasources.

[brancz]

Thanks a lot for putting the work into this!

I'm hesitant about this, as this package was born in a world where Grafana alerting would never be used and all alerting is offloaded to Prometheus/Alertmanager. I feel quite strongly that Grafana alerting in its current form is inferior to that of Prometheus (and Grafana wants to change this but hasn't implemented this yet to my knowledge).

I think I'd prefer if this was just handled within your use of this package and not upstreamed. You should be able to fairly easily add this as a separate patch on top of your stack.

[olivierlemasle]

Thanks @brancz for your answer. I'm ok with keeping this in my installation and not upstreaming it.

I didn't personally compare Prometheus Alertmanager and Grafana alerting (I will!), and I was not responsible for the use of Grafana Alerting in my organization.

However, I feel my use case is not uncommon: using kube-prometheus but having other datasources in Grafana (Azure Monitoring, Elasticsearch, ...) and wanting to use a unified alerting system for all Prometheus datasources.

[brancz]

Just from a reliability perspective it's not a great idea to have Grafana be the thing that delivers your notifications. In the CAP theorem Grafana chooses to be CP (for now, this is what they are trying to change), meaning in the worst case scenario it may not send a notification when it should, while Alertmanager took a different trade-off, which is sending at least one notification (in the worst case multiple).

For other alerting mechanisms the same thing applies. Each of them should produce alerts and fire them against Alertmanager, which has a defined protocol and everything not just Prometheus can potentially speak that protocol.

[ninja-]

But prometheus won't send you a fancy graph with notification :(

[patkoscsaba]

Yeah ... I saw this too late ... I also submitted a pull request for this same thing, only slightly different implementation. https://github.com/brancz/kubernetes-grafana/pull/121 And regarding Grafana vs. Alertmanager ... it depends on what you are monitoring, in what contexts, and who reads the emails. For a technical person, or a devops guy, alertmanager + prometheus is perfect. Ugly emails, lots of technical information, quick route to identify the issue. For a non-technical person, having some of their systems monitored, but if they don't manage the system at a low level, those emails are impossible to comprehend. For example, a notification of "disk is getting full" sent to a small business owner having one server and another few computers must be nice, with graphics, with only a few words. No technical data and low level details are necessary. And no ... creating an email theme and configuring just the right information in it with prometheus + alertmanager is not an easy job, not even for seasoned devops people or software developer. So, yeah, I say each solution has their merits and problems and apply in different contexts.

[ArthurSens]

For a non-technical person, having some of their systems monitored, but if they don't manage the system at a low level, those emails are impossible to comprehend. For example, a notification of "disk is getting full" sent to a small business owner having one server and another few computers must be nice, with graphics, with only a few words. No technical data and low level details are necessary. And no ... creating an email theme and configuring just the right information in it with prometheus + alertmanager is not an easy job, not even for seasoned devops people or software developer. So, yeah, I say each solution has their merits and problems and apply in different contexts.

Since with Grafana Notifications you create alerts tightly coupled with dashboards, you can also create your prometheus alerts tightly coupled with the dashboards that you want by setting Labels/Annotations with a link to the specific dashboard that you want.

Configuring Alertmanager to send alerts with links to the Grafana dashboards is not complicated at all. You can easily provide alerts like this one with the Slack integration for example: