prometheus-msteams icon indicating copy to clipboard operation
prometheus-msteams copied to clipboard

[BUG] Can't send messages to the Power Automate flow

Open savissimo opened this issue 1 year ago • 11 comments

Describe the bug I followed instructions and installed the receiver in my OpenShift cluster. However, I got no messages in Teams (yes, there are alerts firing). I find a stream of these logs in my pod:

2024/05/30 16:14:14 [DEBUG] POST https://prod-251.westeurope.logic.azure.com:443/workflows/<redacted>/triggers/manual/paths/invoke?api-version=2016-06-01&sp=%2Ftriggers%2Fmanual%2Frun&sv=1.0&sig=<redacted>
{"alert":{"receiver":"Critical Teams","status":"firing","alerts":[{"status":"firing","labels":{"alertname":"Outbound errors queue length","instance":"10.128.2.24:8080","job":"outboundlander-service","namespace":"<redacted>","pod":"outboundlander-66fb7b648d-rwdm8","prometheus":"openshift-user-workload-monitoring/user-workload","service":"outboundlander-service","severity":"critical"},"annotations":{"message":"Ci sono molti messaggi nella coda di errori dell'outbound."},"startsAt":"2024-05-30T14:16:50.105Z","endsAt":"0001-01-01T00:00:00Z","generatorURL":"https://thanos-querier-openshift-monitoring.apps.<redacted>/api/graph?g0.expr=rabbitmq_outbound_error_queue_length%7Bjob%3D%22outboundlander-service%22%2Cnamespace%3D%22<redacted>%22%7D+%3E+500&g0.tab=1","fingerprint":"95bf367cd4110cf4"},{"status":"firing","labels":{"alertname":"Outbound errors queue length","instance":"10.128.2.24:8081","job":"outboundlander-service","namespace":"<redacted>","pod":"outboundlander-66fb7b648d-rwdm8","prometh...

Followed by this:

{"caller":"logging.go:25","err":null,"level":"debug","response_message":"{\"error\":{\"code\":\"InvalidRequestContent\",\"message\":\"The input body for trigger 'manual' of type 'Request' must be of type JSON, but was of type 'application/octet-stream'.\"}}","response_status":400,"ts":"2024-05-30T16:16:54.991879121Z","webhook_url":"https://prod-251.westeurope.logic.azure.com:443/workflows/<redacted>/triggers/manual/paths/invoke?api-version=2016-06-01&sp=%2Ftriggers%2Fmanual%2Frun&sv=1.0&sig=<redacted>"}
{"caller":"transport.go:66","host":"promteams.prometheus-msteams.svc.cluster.local:2000","method":"POST","status":200,"took":"71.340822ms","ts":"2024-05-30T16:16:54.991951287Z","uri":"/alertmanager"}

Version Docker image v.1.5.2

To Reproduce I followed the instructions from here: https://examples.openshift.pub/cluster-configuration/monitoring/alertmanager/microsoft-teams-receiver/

Expected behavior The Power Automate flow accepts the request and posts the message.

savissimo avatar May 30 '24 16:05 savissimo

I resolved this with the following:

+++ b/pkg/service/service.go @@ -83,6 +83,8 @@ func (s simpleService) post(ctx context.Context, c card.Office365ConnectorCard, return pr, err }

  •   req.Header.Set("Content-Type", "application/json")
    
  •   resp, err := s.client.Do(req)
      if err != nil {
              err = fmt.Errorf("http client failed: %w", err)
    

tekphobia avatar Jul 08 '24 19:07 tekphobia

Great news, thank you! Will there be a Docker release for this fix?

savissimo avatar Jul 08 '24 21:07 savissimo

Just for info: Microsoft is retiring the Office 365 connectors soon. Official announcement: https://devblogs.microsoft.com/microsoft365dev/retirement-of-office-365-connectors-within-microsoft-teams/

This bugfix is needed to switch to the Power Automate workflows.

MS Teams has started to add the following text when you use the old API:

Action Required:
O365 connectors within Teams will be deprecated and notifications from this service will stop.
[Learn more](https://aka.ms/O365ConnectorDeprecation)
about the timing and how the Workflows app provides a more flexible and secure experience. If you want to continue receiving these types of messages, you can use a workflow to post messages from a webhook request.

spranta-devops avatar Jul 09 '24 07:07 spranta-devops

In fact I used the service in conjunction with a Power Automate flow. This may be the reason for the bug.

simonesaviolo-reda avatar Jul 09 '24 10:07 simonesaviolo-reda

Great news, thank you! Will there be a Docker release for this fix?

I would imagine once the PR is merged you will be able to build the docker image with: make docker

tekphobia avatar Jul 09 '24 13:07 tekphobia

Should I push that image to a private repository of mine or will it be pushed to the project's repository on quay.io?

savissimo avatar Jul 09 '24 13:07 savissimo

Hi :) Has anyone been able to use Power Automate flows after changing the line of code to send as application/json ?

So far here, I keep getting errors because the format expected by Power Automate is different (seems they only support Adaptive Cards being sent through webhooks ?)

Unyxos avatar Jul 18 '24 08:07 Unyxos

FWIW, here are some additional references that you may find useful:

  • https://gist.github.com/chusiang/895f6406fbf9285c58ad0a3ace13d025
    • lots of discussion & an example script to use adaptive cards format
  • https://github.com/tlolkema/simple-teams-message
    • the action.yml file contains an example payload that could be adapted for use here

atc0005 avatar Jul 18 '24 10:07 atc0005

thanks for providing this msteams proxy. it pretty did his job until now. but with the power automate change it would be nice to have the ContentType Header fixed or configureable.

So thanks for fixing this :)

sbickmann avatar Jan 13 '25 12:01 sbickmann

I believe an MS Teams integration was added directly into Alert Manager supporting Flows (not tested yet on my side) image

https://github.com/prometheus/alertmanager/pull/4024

pbenefice avatar Jan 14 '25 10:01 pbenefice

I believe an MS Teams integration was added directly into Alert Manager supporting Flows (not tested yet on my side)

prometheus/alertmanager#4024

That's true, but some users reported that the new integration offers much less flexibility than this service. It would be great to be able to choose what to use.

savissimo avatar Jan 14 '25 10:01 savissimo