alertmanager-discord icon indicating copy to clipboard operation
alertmanager-discord copied to clipboard

Failed to unpack inbound alert request

Open acmodeu opened this issue 2 years ago • 4 comments

I want to send some alerts from external exporter via kubernetes prometheus stack to discord. I see the alerts in the alertmanager but they don't appear in discord channel and there are a lot of similar errors in alertmanager-discord logs.

2021/08/12 19:46:39 10.200.129.205:9094 - [GET] 
2021/08/12 19:46:39 Failed to unpack inbound alert request - 

acmodeu avatar Aug 12 '21 19:08 acmodeu

Are you sure the alert manager is actually sending the alert webhooks to this software?

Those logs you provided look like they are just health checks, presumably from k8s On Thu, 12 Aug 2021, 20:50 acmodeu, @.***> wrote:

I want to send some alerts from external exporter via kubernetes prometheus stack to discord. I see the alerts in the alertmanager but they don't appear in discord channel and there are a lot of similar errors in alertmanager-discord logs.

2021/08/12 19:46:39 10.200.129.205:9094 - [GET] 2021/08/12 19:46:39 Failed to unpack inbound alert request -

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/benjojo/alertmanager-discord/issues/24, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALPK4WM2MEKBMHAEGEDJHDT4QQX5ANCNFSM5CCARETA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

benjojo avatar Aug 12 '21 20:08 benjojo

Yes. I'm sure, I set up discord as default receiver for all alerts but received all other kubernetes alerts except those external ones. I switched to telegram bot and received them right away.

acmodeu avatar Aug 12 '21 21:08 acmodeu

So, alertmanager webhooks come in as POST's so you should look for something like

2021/08/12 19:46:39 10.200.129.205:9094 - [POST] 

in your logs, if you can't find them, webhooks are not directly landing on alertmanager-discord.

benjojo avatar Aug 12 '21 21:08 benjojo

Hi,

When using into k8s, you may want to use readiness and liveness probes. They usually make GET requests to the app, to see if the process is alive and fit for duty. If something other than 2xx is returned, kubernetes will kill the container, and spawn another one.

h4wkmoon avatar Aug 23 '21 13:08 h4wkmoon