alertmanager icon indicating copy to clipboard operation
alertmanager copied to clipboard

Proposal: improve silence API

Open Tontaube opened this issue 1 year ago • 2 comments

What did you do? I add silence for too noisy alerts already being open incidents in our incident management system. I added an identifier (e.g. ticket nr) from the incident management system as author/ createdBy or to the comment.

What did you expect to see? I like the incident management system to be able to query silences for a specific createdBy content or even for pattern in the comment to check if the silence still exists (as it needs to be cleaned up before closing an incident).

What did you see instead? Under which circumstances? Silences can be filtered by matching labels only.

Environment

not relevant

Proposal on /silences GET route, there should be implemented additional filter parameters for the createdBy and comment field not for the matchers only. Additionally, a filter for active silences may make sense.

Alternative There could also be done separate full-fledged proposal proposing an additional "labels" data structure on silences, which would avoid any regex searches on the comment field. (not to be mixed up with the alert labels, here those would be labels set by the user / receiving systems on the silence entry).

Goals

  • make it possible to receive a filtered list of Silences on silence-specific fields like createdBy or comment

Audience Everyone using alertmanager together with an incident mgmt system and going into automation.

Possible action plan

  • adapt openapi spec
  • implement additional filters
  • optional: add gui functionality

Resources We might be able to spend some resources on it, for sure on testing side, to support a timeline of having this capability merged until end of July.

References

  • alertmanager openapi v2 spec: https://github.com/prometheus/alertmanager/blob/main/api/v2/openapi.yaml

I start lightweight and will enrich this issue with results from discussion in #[email protected]

Tontaube avatar Jun 02 '23 09:06 Tontaube