robusta icon indicating copy to clipboard operation
robusta copied to clipboard

Resolved Alert Handling

Open vmusaely opened this issue 3 years ago • 1 comments
trafficstars

I've pushed kill_pod action to robusta runner pod, when I'm getting crashed pod it works fine and deletes that pod, after that the new one is created. But it runs the action even on messages that say “[RESOLVED]”, and deletes new created healthy pod.

vmusaely avatar May 24 '22 16:05 vmusaely

Hi @vmusaely ,

I think you can achieve what you need by filtering the alert trigger only on firing alerts (and not resolved)

Your playbook configuration in your values.yaml file should be:

- triggers:
  - on_prometheus_alert:
      alert_name: YOUR CRASHING POD ALERT NAME
      status: "firing"
  actions:
  - kill_pod: {}

Does that work for you?

arikalon1 avatar May 25 '22 10:05 arikalon1

@arikalon1 Did we change the behaviour on this so that on_prometheus_alert only fires on triggered alerts by default?

If so, we can close this.

aantn avatar Aug 19 '22 12:08 aantn

Yes, we did. on_prometheus_alert is triggered, by default, only for firing alerts. Closing it

arikalon1 avatar Aug 19 '22 14:08 arikalon1