alloy
alloy copied to clipboard
[loki.rules.kubernetes] pipelines in the expression
What's wrong?
It's not possible to use the loki.rules.kubernetes component to add rules to the loki ruler using the promehteusrule object. expressions to filter logs often use | symbols, but it's giving issues for this component.
Steps to reproduce
I created a prometheusrule object to test alerting on loki.
The following spec on the prometheusrule works:
spec:
groups:
- name: rabbitmqtest.alerts
rules:
- alert: alottofrabbitmqerrorsapplication
annotations:
description: High error rate on rabbitmq errors
summary: High error rate on rabbitmq errors
expr: sum(rate({app=~"application.*"} [5m]))
for: 5m
But since I'd like to filter out not all logs, but only logs with the level error the expression is the following:
expr: sum(rate({app=~"application.*"} | json | Level="Error" [5m]))
This give the error saying that | is not allowed:
Failed to save resource: {"error":{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"admission webhook "prometheusrulemutate.monitoring.coreos.com" denied the request: Rules are not valid","reason":"Invalid","details":{"name":"prometheusrules","causes":[{"message":"8:11: group "rabbitmqtest.alerts", rule 1, "alottofrabbitmqerrorsapplication": could not parse expression: 1:33: parse error: unexpected character: '|'"}]},"code":422},"messages":["admission webhook "prometheusrulemutate.monitoring.coreos.com" denied the request: Rules are not valid"],"isUsedForNotification":false}
Changing the expr to:
expr: |-
'sum(rate({app=~"application.*"} | json | Level="Error" [5m]))'
makes it possible to complete the creation of the prometheusrule object but the alloy give the following error: server returned HTTP status 400 Bad Request: could not parse expression for alert 'alottofrabbitmqerrorsapplication' in group 'rabbitmqtest.alerts': parse error at line 1, col 1: invalid char literal\n\n
Is there a way to work with expressions with pipelines? Maybe the crd of prometheus rule should be addapted? or can we implement a work around?
Thanks for checkng
System information
kubernetes
Software version
Grafana Alloy v1.0.0
Configuration
No response
Logs
No response