prometheus-engine
prometheus-engine copied to clipboard
Alerting labels
We are trying to use the alerting labels. This is port of the existing Prometheus alert
- alert: KubernetesNodeReady
expr: kube_node_status_condition{condition="Ready",status="true"} == 0
for: 10m
labels:
severity: critical
annotations:
summary: Kubernetes Node ready (instance {{ $labels.instance }})
description: "Node {{ $labels.node }} has been unready for a long time\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"
We ran into this error Error: UPGRADE FAILED: parse error at (gmp/templates/alerts/k8s.yaml:18): undefined variable "$labels"
How do we access labels
?
cc @lyanco
We're not sure - we think this should work. Try the suggestion here perhaps? https://stackoverflow.com/questions/64693812/how-to-fix-the-error-undefined-variable-labelsin-prometheus/64694592#64694592
We're not sure - we think this should work. Try the suggestion here perhaps? https://stackoverflow.com/questions/64693812/how-to-fix-the-error-undefined-variable-labelsin-prometheus/64694592#64694592
Could you please show an example that uses label
that should work?
- record: job_up_sum
expr: sum by(job, project_id) (up)
- alert: TooFewReplicas
expr: job_up_sum < 3
for: 1m
annotations:
description: "Not enough replicas ({{ $value }}) for job {{ $labels.job }}"
I also added your rules and it worked. I think something might be to your templating.
If what you are trying to do is pass a label from the query triggering the alert to the alert being passed to alertmanager, try this:
kind: PrometheusRule
.......
labels:
instance: '{{`{{`}} $labels.instance{{`}}`}}'
This only worked after adding both single quotes and the rendering of curly brackets in helm.