global alerting rule configuration

[parkedwards]

hey there!

we currently use sloth to configure a number of SLOs. currently, we specify alerting: configurations for each SLO, but they're all the same (same name, description text, etc.)

is there any way to specify a global alerting configuration?

[r3code]

Why they all the same? Can you show some examples?

[parkedwards]

@r3code our alerting config looks something like this:

    alerting:
      name: SLOBurnRateTooHigh
      labels:
        custom_slack_channel: "slo-multiwindow-burn-rate-alerts"
      annotations:
        description: '`{{$value | humanizePercentage}}` error rate for `{{$labels.sloth_id}}`'
        dashboard: "https://grafana.company.org?service={{$labels.sloth_service}}&var-slo={{$labels.sloth_slo}}"
      page_alert:
        labels:
          severity: warning
        annotations:
          impact: '[high] `>= 2%` of monthly error budget consumed in the last `1h-6h`'
      ticket_alert:
        labels:
          severity: info
        annotations:
          impact: '[moderate] `>= 10%` of monthly error budget consumed in the last `1d-3d`'

and then our alert will collect all firing events and list them in the message, so we can see which SLOs are triggering whichever alert

[r3code]

I see. I'm struggling the same and the only way I see now is to create a jsonnet wrapper over sloth specs and generate specs from jsonnet slo specs. Or maybe it worth to extend the openslo spec support as it supports multiple alert conditions and refs to reuse them. Also they now support composite SLO, which allows you to have an aggregate SLO, continuing a sum weighted SLOs.

[parkedwards]

ahh interesting. yeah im avoiding jsonnet as much as possible, but maybe that's the only way to customize further