grafana icon indicating copy to clipboard operation
grafana copied to clipboard

Alerting: `/api/prometheus/grafana/api/v1/alerts` uses invalid state names

Open piotrp opened this issue 3 years ago • 5 comments

/api/prometheus/grafana/api/v1/alerts endpoints returns alert's state as Normal/Pending/Alerting, instead of inactive/pending/firing as required by https://github.com/prometheus/compliance/blob/main/alert_generator/specification.md.

Looks like Grafana passes string value of its internal alert state, without any translation: https://github.com/grafana/grafana/blob/623255cf2d5e34e2b87308d70bf60a06e7e70080/pkg/services/ngalert/api/api_prometheus.go#L61 https://github.com/grafana/grafana/blob/623255cf2d5e34e2b87308d70bf60a06e7e70080/pkg/services/ngalert/eval/eval.go#L111-L134

Tested on Grafana 8.5.6.

piotrp avatar Jul 19 '22 12:07 piotrp

After looking at this a bit longer I think it's by design, and this endpoint is only inspired by Alert Generator spec, and doesn't try to follow it

piotrp avatar Jul 20 '22 22:07 piotrp

Thanks for reaching out @piotrp, we already run the alert generator spec against grafana alerting and we are not fully compliant yet. Our goal is to become eventually compliant.

I will keep this issue open to track this part of the effort.

JohnnyQQQQ avatar Jul 21 '22 09:07 JohnnyQQQQ

Thank you, it's good to know where this is going. I'm writing custom integration that puts all active alerts into a table and sends it via email, so I'm looking closely at what is available in current APIs, and what can be inferred by looking at available annotations/labels.

piotrp avatar Jul 21 '22 10:07 piotrp

Small update, for completeness sake:

  • health - error is reported by "error" instead of "err" (plus there's additional "nodata" health state)
  • there's also "NoData" alert state

For my use it would be ideal to have additional Grafana-specific information present in additional attributes (or private annotations?):

  • explicit nodata indicator (because compliance means that this will disappear from places where it's currently present, and would need to be inferred from present labels)
  • multiple values coming from firing alert with classic condition, here even having __value_string__ annotation, like in /alertmanager, would work as this can be parsed to extract metric names and values
  • some way to tell whether alert is silenced

piotrp avatar Jul 22 '22 12:07 piotrp

This issue has been automatically marked as stale because it has not had activity in the last year. It will be closed in 30 days if no further activity occurs. Please feel free to leave a comment if you believe the issue is still relevant. Thank you for your contributions!

github-actions[bot] avatar May 04 '24 01:05 github-actions[bot]

This issue has been automatically closed because it has not had any further activity in the last 30 days. Thank you for your contributions!

github-actions[bot] avatar Jun 03 '24 01:06 github-actions[bot]