Silencing alerts in robusta config not working

[omfurman]

I went with the docs - https://docs.robusta.dev/master/catalog/actions/event-enrichment.html?highlight=silence and added some alerts I would like to silence in the values.yaml of the installation like this:

customPlaybooks:
- triggers:
  - on_prometheus_alert: {}
  actions:
  - name_silencer: 
      names:
      - "CPUThrottlingHigh"

It seemed to be working, but I'm getting alerts like this all the time: Am I missing something?

[Avi-Robusta]

Hey, Can you run: robusta logs --namespace NAMESPACE_HERE | grep -in CPUThrottlingHigh Im expecting there to be some logs that say "silencing alert CPUThrottlingHigh"

[omfurman]

Sure, its returning the following output:

195:2022-05-22 13:55:43.184 INFO     Successfully loaded Kubernetes resource storybook-xxx for alert CPUThrottlingHigh
196:2022-05-22 13:55:43.200 INFO     Successfully loaded Kubernetes resource storybook-xxxx for alert CPUThrottlingHigh ```

[Avi-Robusta]

Hey @omfurman , can you quickly run robusta playbooks edit-config, look on active_playbooks (and customPlaybooks) section, and see if you can spot your customPlaybook in there, and where is it located in relation to the other playbooks?

I'm thinking maybe we have some ordering issue where your playbook is placed after the alert reporting playbooks

[omfurman]

@Avi-Robusta Its located at the very end of the list and the config file, I can also see its after a CpuThrottling action:

 - actions:
    - cpu_throttling_analysis_enricher: {}
    - alert_graph_enricher:
        item_type: Pod
        resource_type: CPU
    triggers:
    - on_prometheus_alert:
        alert_name: CPUThrottlingHigh
  - actions:
    - deployment_status_enricher: {}
    triggers:
    - on_prometheus_alert:
        alert_name: KubernetesDeploymentReplicasMismatch
  - actions:
    - default_enricher: {}
    triggers:
    - on_prometheus_alert: {}
  - actions:
    - name_silencer:
        names:
        - CPUThrottlingHigh
    triggers:
    - on_prometheus_alert: {}
    ```

[Avi-Robusta]

@omfurman at the top of the config there will be a builtin playbook that looks like this

>   - actions:
>     - name_silencer:
>         names:
>         - Watchdog
>         - KubeSchedulerDown
>         - KubeControllerManagerDown
>         - InfoInhibitor
>     triggers:
>     - on_prometheus_alert: {}

I would recommend adding your silencers to this list in the meantime. We will come up with a solution on our end shortly and alert you know when we fixed the issue.

[Avi-Robusta]

Hi @omfurman, We are pushing a fix now, it will be in the next robusta release. https://github.com/robusta-dev/robusta/pull/342/files Thanks for letting us know about this issue!

[aantn]

Closing, as I believe this is fixed. Feel free to re-open if there are any more issues.