Add Prometheus Metrics

[abhishekjiitr]

Closes #2026 :clinking_glasses:

Start exporting metrics in Prometheus format on port 9090. :chart_with_upwards_trend:

Exposing metrics helps in monitoring ElastAlert itself, which can be further used to generate alerts in case something goes wrong (e.g. ElastAlert is not able to send alerts or encountering errors). :pager:

This may be a important feature for those who are running business critical rules in ElastAlert and need to act quickly if ElastAlert is not functioning as expected.

Relevant Metrics added, most of the metrics have a rule_name_label:

1. Number of hits - Per Rule
2. Number of matches - Per Rule
3. :watch: Time taken to evaluate rule - Per Rule
4. Number of alerts sent - Per Rule
5. Number of alerts not sent- Per Rule
6. Total Number of elastalert_errors - Global
7. Number of silenced alerts - Per Rule

[abhishekjiitr]

@pdscopes @Qmando @danielpops can you take a look?

[pdscopes]

Firstly, this could be converted into a script that scrapeselastalert_alert, elastalert_status, etc rather than needing to be integrated directly into ElastAlert.

Secondly, you should probably create a wrapper class and move the metrics definitions and update_metrics into said class.

Finally, you've not provided any way of configuring/turning off the Prometheus nor provided any documentation.

[abbasalizaidi]

@abhishekjiitr I am looking for something similar to this. Would be great if it can be merged.

I will be happy to do the documentation part!

@Qmando Your thoughts?