robusta
robusta copied to clipboard
feat: Enable per-alert Incident Severity
Changes
This PR improves on existing Incident.io implementation and allows to define per-alert incident severity in Incident.io (until now severity was determined by the alert severity, however not all alerts with the critical severity should have the same Incident.io severity).
The following configurational changes were made:
severity_alert_label_name- label on the alert that will contain information about Incident.io severity. Default set torobusta_incidentio_severity.severity_default- sane default in case when label is not defineddashboard_url_annotation_name- Incident.io has something called thesource_url. Usually, that can be a link toward Grafana dashboard or anything else. In this case, it is a name of the alert annotation. Default set todashboard_url.runbook_url_annotation_name- same as with the Dashboard URL, stored as metadata information that can be extracted and parsed in Incident.io interface.
Please note that severity names must be mapped in the Incident.io interface (whatever you pass - minor, major, critical) to the actual severity ID.
Risks
None
Performance impact
None
Security impact
None
How to QA
- Configure Incident.io Sink
- Send a few simulated requests
./robusta playbooks trigger prometheus_alert --namespace=monitoring alert_name="IncidentIoSeverityTesting" severity=critical labels="robusta_incidentio_severity:major"
./robusta playbooks trigger prometheus_alert --namespace=monitoring alert_name="IncidentIoSeverityTesting" severity=critical labels="robusta_incidentio_severity:major" annotations="dashboard_url=https://grafana.com,runbook_url=https://google.com"
- Go to Incident.io UI, find alerts and inspect payload