robusta icon indicating copy to clipboard operation
robusta copied to clipboard

feat: Enable per-alert Incident Severity

Open igorhrcek opened this issue 11 months ago • 0 comments

Changes

This PR improves on existing Incident.io implementation and allows to define per-alert incident severity in Incident.io (until now severity was determined by the alert severity, however not all alerts with the critical severity should have the same Incident.io severity).

The following configurational changes were made:

  1. severity_alert_label_name - label on the alert that will contain information about Incident.io severity. Default set to robusta_incidentio_severity.
  2. severity_default - sane default in case when label is not defined
  3. dashboard_url_annotation_name - Incident.io has something called the source_url. Usually, that can be a link toward Grafana dashboard or anything else. In this case, it is a name of the alert annotation. Default set to dashboard_url.
  4. runbook_url_annotation_name - same as with the Dashboard URL, stored as metadata information that can be extracted and parsed in Incident.io interface.

Please note that severity names must be mapped in the Incident.io interface (whatever you pass - minor, major, critical) to the actual severity ID.

Risks

None

Performance impact

None

Security impact

None

How to QA

  1. Configure Incident.io Sink
  2. Send a few simulated requests
./robusta playbooks trigger prometheus_alert --namespace=monitoring alert_name="IncidentIoSeverityTesting" severity=critical labels="robusta_incidentio_severity:major"
./robusta playbooks trigger prometheus_alert --namespace=monitoring alert_name="IncidentIoSeverityTesting" severity=critical labels="robusta_incidentio_severity:major" annotations="dashboard_url=https://grafana.com,runbook_url=https://google.com"
  1. Go to Incident.io UI, find alerts and inspect payload

igorhrcek avatar Jan 21 '25 15:01 igorhrcek