robusta
robusta copied to clipboard
Using alert fields in alert_handling_job
Is your feature request related to a problem? Please describe. When running jobs on prometheus triggers user should be able to use alert fields in generated-values.yaml file in job command. Also user should be able to setup their own environment variables for job.
Describe the solution you'd like Instead of providing the alert params to job directly. What we can do is provide the alert params to user so it can be used in generated-values.yaml file in job command. It would allow the user flexibility to set up their own environment variables for the job in addition to alert params.
Describe alternatives you've considered Right now the alternative I have considered is to write my own code, make a docker image of it and push it to dockerhub to use it as a job.
Additional context Below is a sample code as to how user would be able to use the alert params.
- triggers:
- on_prometheus_alert:
alert_name: KubePodCrashLooping
actions:
- alert_handling_job:
command:
- 'kubectl'
- 'delete'
- 'pod'
- ${ALERT_OBJ_NAME} #Using alert param
- '--force'
- '--grace-period=0'
image: bitnami/kubectl
service_account: deleting-pods
job_ttl_after_finished: 10
notify: true
name: delete-pod-job
namespace: default
backoff_limit: 2
By the way, I think that's a really cool example to include in the docs! As soon as we get this implemented, I'd love to add an example like that to https://docs.robusta.dev/master/tutorials/prometheus-enrichment.html
This is now supported :)
https://docs.robusta.dev/master/playbook-reference/actions/remediation.html#remediation