prometheus
[WIP] Validate notifier configurations statically.
Resolves #3490.
As discussed in #3490, notifiers may have invalid configuration, including but not limited to invalid inline templates, that isn't detected until alertmanager attempts to send a notification. At that point, alertmanager drops alerts. This patch introduces a pattern, implemented so far for a few notifiers, of validating notifier configuration in the notifier constructor.
We discussed in #3490 whether we should add validation for notifiers one by one or all at once. As of this writing, I've drafted validation for the opsgenie and slack receivers, and you can see that both were pretty simple to implement, since we can mostly reuse the code that actually sends the notifications—specifically, we can build a request using an empty alert and check for errors without actually sending a notification. If we want to implement validation for all notifiers before merging any changes, I'm happy to continue working through the list of notifiers.
cc @grobinson-grafana @simonpasquier @gotjosh
I noticed that this might overlap with #3491 from @alexweav and @OktarianTB. Let me know if this is redundant! Although I think I still like the idea of being able to validate receiver/notifier configs statically—for example, I would probably prefer static validation over sending a real notification in a CI environment, which I don't want to depend on third-party APIs.
