helm-charts
helm-charts copied to clipboard
prometheus-community
[kube-prometheus-stack] Unable to get Grafana alerts working
Describe the bug a clear and concise description of what the bug is.
Unable to get any Grafana alerts sent to contact points. Create both a Slack and MS Teams contact point, set one at the root policy and created a alert rule.
Nothing gets sent to contact points.
What's your helm version?
version.BuildInfo{Version:"v3.7.0", GitCommit:"eeac83883cb4014fe60267ec6373570374ce770b", GitTreeState:"clean", GoVersion:"go1.16.8"}
What's your kubectl version?
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:31:21Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"092fbfbf53427de67cac1e9fa54aaa09a28371d7", GitTreeState:"clean", BuildDate:"2021-06-16T12:53:14Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}
Which chart?
kube-prometheus-stack
What's the chart version?
19.0.1
What happened?
Nothing is sent to my contact points
What you expected to happen?
Alerts are sent to contact points
How to reproduce it?
- Create a Slack contact point with a webhook
- Make that contact point the root policy one and group by
alertname - Create an alert rule with
vector(1)> 0
Enter the changed values of values.yaml?
grafana:
grafana.ini:
feature_toggles:
enable: ngalert
Enter the command that you execute and failing/misfunctioning.
minikube start --kubernetes-version=v1.21.2
helm install -f value.yml prom prometheus/kube-prometheus-stack
Anything else we need to know?
No response
@mymasse it's a bug in Grafana < 8.1.5. I'm also waiting for the chart to update to lastest grafana to test if it's fixed
Seems like the newer chart fixed part of the problem. Any Grafana defined alerts now are sent to contact points. However all the alerts in the Cortex/Loki group are still not working, for instance these:
These I think came pre-bundled with the chart, will show up in Alert-manager but other that the Alert Rules panel do not show up in the Alerts Groups when they are in the Firing state and are not propagated to contact points.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.
![stale[bot] avatar](https://avatars.githubusercontent.com/in/1724?v=4 "Published by a pub.dev verified publisher"){kind=small}
Thanks for raising this @mymasse.
I can confirm the Loki generated alerts unfortunately still don't go through to the defined Contact Point (in my case, Alertmanager) as of latest chart version 23.1.6.
Hope this gets a resolution soon, it has been quite a few chart version since the bug's been opened... and it can be a bit of a roadblock.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.
I am also interested in the resolution of this issue. Please reopen again.
The versions in our system are: kube-prometheus-stack: 32.2.0 prometheus: 2.33.1 grafana: 8.3.6 alertmanager: 0.23.0 prometheus-operator: 0.54.0
Plus 1 for resolution. Is there a workaround or something ? Using the latest version.
Found this statement on this issue from one of the Grafana team members: https://github.com/grafana/grafana/discussions/45773#discussioncomment-2235421
tldr: Not possible atm, but might be in the future :(
+1
I am also searching for a resolution. I'm using the latest version
Could it be that these alerts are managed by the alertmanager? My alertmanager shows Cluster Status: disabled under http://localhost:9093/#/status
I am not able to enable this, I have found the following stackoverflow question describing my issue, but I don't know where I can enable this option.
https://stackoverflow.com/questions/70470152/alertmanager-cluster-status-is-disabled
If this is unrelated I can open a new issue for this.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.
I'm also looking for a clever solution to this issue. Is this related to the fact that Grafana's Alertmanager doesn't receive alerts from Loki as described here:
https://grafana.com/docs/grafana/latest/alerting/fundamentals/alertmanager/
Can you please reopen this @mrueg?
Is this also related to: https://github.com/grafana/grafana/issues/39219?
Actually reading through https://github.com/grafana/grafana/issues/39219 this is exactly the same issue. I think it should remain closed. Refer to the closing comment on the Grafana issue for details.