alloy icon indicating copy to clipboard operation
alloy copied to clipboard

[loki.rules.kubernetes] pipelines in the expression

Open ToonTijtgat2 opened this issue 9 months ago • 1 comments

What's wrong?

It's not possible to use the loki.rules.kubernetes component to add rules to the loki ruler using the promehteusrule object. expressions to filter logs often use | symbols, but it's giving issues for this component.

Steps to reproduce

I created a prometheusrule object to test alerting on loki.

The following spec on the prometheusrule works:

spec:
  groups:
    - name: rabbitmqtest.alerts
      rules:
        - alert: alottofrabbitmqerrorsapplication
          annotations:
            description: High error rate on rabbitmq errors
            summary: High error rate on rabbitmq errors
          expr: sum(rate({app=~"application.*"} [5m]))
          for: 5m

But since I'd like to filter out not all logs, but only logs with the level error the expression is the following: expr: sum(rate({app=~"application.*"} | json | Level="Error" [5m])) This give the error saying that | is not allowed: Failed to save resource: {"error":{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"admission webhook "prometheusrulemutate.monitoring.coreos.com" denied the request: Rules are not valid","reason":"Invalid","details":{"name":"prometheusrules","causes":[{"message":"8:11: group "rabbitmqtest.alerts", rule 1, "alottofrabbitmqerrorsapplication": could not parse expression: 1:33: parse error: unexpected character: '|'"}]},"code":422},"messages":["admission webhook "prometheusrulemutate.monitoring.coreos.com" denied the request: Rules are not valid"],"isUsedForNotification":false}

Changing the expr to:

expr: |-
            'sum(rate({app=~"application.*"} | json | Level="Error" [5m]))'

makes it possible to complete the creation of the prometheusrule object but the alloy give the following error: server returned HTTP status 400 Bad Request: could not parse expression for alert 'alottofrabbitmqerrorsapplication' in group 'rabbitmqtest.alerts': parse error at line 1, col 1: invalid char literal\n\n

Is there a way to work with expressions with pipelines? Maybe the crd of prometheus rule should be addapted? or can we implement a work around?

Thanks for checkng

System information

kubernetes

Software version

Grafana Alloy v1.0.0

Configuration

No response

Logs

No response

ToonTijtgat2 avatar May 06 '24 08:05 ToonTijtgat2