robusta icon indicating copy to clipboard operation
robusta copied to clipboard

Using alert fields in alert_handling_job

Open alikhanxgrid opened this issue 2 years ago • 1 comments

Is your feature request related to a problem? Please describe. When running jobs on prometheus triggers user should be able to use alert fields in generated-values.yaml file in job command. Also user should be able to setup their own environment variables for job.

Describe the solution you'd like Instead of providing the alert params to job directly. What we can do is provide the alert params to user so it can be used in generated-values.yaml file in job command. It would allow the user flexibility to set up their own environment variables for the job in addition to alert params.

Describe alternatives you've considered Right now the alternative I have considered is to write my own code, make a docker image of it and push it to dockerhub to use it as a job.

Additional context Below is a sample code as to how user would be able to use the alert params.

- triggers:
  - on_prometheus_alert:
     alert_name: KubePodCrashLooping
  actions:
    - alert_handling_job:
       command:
       - 'kubectl'
       - 'delete'
       - 'pod'
       - ${ALERT_OBJ_NAME} #Using alert param 
       - '--force'
       - '--grace-period=0'
       image: bitnami/kubectl
       service_account: deleting-pods
       job_ttl_after_finished: 10
       notify: true
       name: delete-pod-job
       namespace: default
       backoff_limit: 2

alikhanxgrid avatar Jul 21 '22 13:07 alikhanxgrid

By the way, I think that's a really cool example to include in the docs! As soon as we get this implemented, I'd love to add an example like that to https://docs.robusta.dev/master/tutorials/prometheus-enrichment.html

aantn avatar Jul 21 '22 15:07 aantn

This is now supported :)

https://docs.robusta.dev/master/playbook-reference/actions/remediation.html#remediation

aantn avatar Feb 22 '24 16:02 aantn