issue #6667 implement support for mimir.rules.kubernetes federated group rules

[GroovyCarrot]

Federated group rules are loaded from the monitoring.grafana.com/source_tenants annotation on the prometheusrules crd.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: tenant1-group1
  annotations:
    monitoring.grafana.com/source_tenants: tenant1,tenant2
spec:
  groups:
  - ...

PR Description

Which issue(s) this PR fixes

Fixes grafana/alloy#216

Notes to the Reviewer

The mimir rule group != prometheus rule group due to the source_tenants field. I had to use a new type and embedded rulefmt.RuleGroup in internal/mimir/client/types.go

This approach seemed the most flexible, given the field will add query load to the other tenants. Either way the type change is necessary to send the field to mimir, happy to discuss alternatives.

PR Checklist

• [ ] CHANGELOG.md updated
• [ ] Documentation added
• [x] Tests updated
• [ ] Config converters updated

[CLAassistant]

All committers have signed the CLA.

[GroovyCarrot]

I've build the image and deployed it to k8s cluster to test the annotation is correctly found, and the resource written to mimir ruler by hitting /ruler/rule_groups.

river config:

mimir.rules.kubernetes "tenant1" {
  address = "http://mimir-ruler:8080"
  tenant_id = "tenant1"

  rule_namespace_selector {
    match_labels = {
      "kubernetes.io/metadata.name" = "monitoring",
    }
  }

  rule_selector {
    match_labels = {
      "tenant" = "tenant1",
    }
  }
}

prometheus rule crd:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: test
  annotations:
    monitoring.grafana.com/source_tenants: "tenant2"
  labels:
    tenant: "tenant1"
spec:
  groups:
  - name: group1
    rules:
    - record: test_record
      expr: sum by (label) (increase(something[1m]))
      labels:
        test: test
    - alert: test_alert
      annotations:
        summary: test alert
      expr: sum by (label) (increase(something[1m])) > 0
      labels:
        test: test

agent logs:

ts=2024-03-13T10:51:10.423943177Z level=info msg="processing event" component=mimir.rules.kubernetes.tenant1 type=resource-changed key=monitoring/test
ts=2024-03-13T10:51:12.917942128Z level=info msg="added rule group" component=mimir.rules.kubernetes.tenant1 namespace=agent/monitoring/test/a08ba9e4-0555-4434-8231-d9b8299063a4 group=group1

confirm with ruler:

$ curl http://localhost:8080/ruler/rule_groups

tenant1:
    agent/monitoring/test/a08ba9e4-0555-4434-8231-d9b8299063a4:
        - name: group1
          rules:
            - record: test_record
              expr: sum by (label) (increase(something[1m]))
              labels:
                test: test
            - alert: test_alert
              expr: sum by (label) (increase(something[1m])) > 0
              labels:
                test: test
              annotations:
                summary: test alert
          source_tenants: # correctly set
            - tenant2

[jeremych1000]

Could we get a pair of eyes on this? :eyes: This is a blocker in our multi-tenant environment atm :)

[bcrisp4]

Does this PR need to be opened on grafana/alloy now perhaps? I'm also keen to get this feature

[GroovyCarrot]

have rebased the PR; any chance we can get some eyes on it?

[github-actions[bot]]

This PR has not had any activity in the past 30 days, so the needs-attention label has been added to it. If you do not have enough time to follow up on this PR or you think it's no longer relevant, consider closing it. The needs-attention label signals to maintainers that something has fallen through the cracks. No action is needed by you; your PR will be kept open and you do not have to respond to this comment. The label will be removed the next time this job runs if there is new activity. Thank you for your contributions!