alertmanager
alertmanager copied to clipboard
prometheus
[Feature] Support adaptive cards for MSTeams
Hi there,
I saw that MS Teams support was recently added, many thanks for that! I've been using prometheus-msteams at work for quite some time and it will be great to have the integration natively.
It would be awesome if support for the newer adaptive cards message format could be added at some point.
Here's the payload needed to fire these into Teams:
{
"type":"message",
"attachments":[
{
"contentType":"application/vnd.microsoft.card.adaptive",
"contentUrl":null,
"content":{
<user provided JSON for adaptive card goes here>
}
}
]
}
And here's a sample alert that I created: https://gist.github.com/aw1cks/20c60986e789342a5e1e847b5d05a954
Proposal
Add a configuration option for the msteams integration, to switch the payload format to adaptive web cards, delegating responsibility for creating valid adaptive card JSON to the user's template.
One complication is that it's not trivial to keep a separate title and text template per current implementation - but I'd propose that if the aforementioned config option were enabled, that it'd be left to the user to template the title as they saw fit, and just read the text field, placing it directly into the content field of the above JSON payload.
If such an approach would be accepted, I'd be happy to look at doing this!
I did play around with this - here's a somewhat hacky, but working change: https://github.com/aw1cks-forks/alertmanager/commit/5fca6eb4f68b516d52b13975e914ce12c08ae48d I'd imagine it'd be best to keep the existing MessageCard implementation and allow users to opt into adaptive cards if desired.
I added support for Adaptive Cards to Grafana. Perhaps we can use the same code for Alertmanager too? @gotjosh
Nice! That seems like a path of low resistance.
My only comment is that the card format appears to be quite prescriptive, which does limit the cusomisability.
@aw1cks Been playing around with this today. Got some nice dynamic creation of the Cards to work well all through just passing in the JSON data inside a .tmpl file that was defining the text. Awesome stuff.
Can you consider opening a PR for your changes? It makes the alerting experience in Teams much better.
Firing
Resolved
Label / Annotations View:
Silences:
{{ define "new.text" }}
{
"$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
"type": "AdaptiveCard",
"version": "1.2",
"padding": "None",
"msteams": {
"width": "Full"
},
"body": [
{
"type": "Container",
"id": "alert-msg",
"padding": "Default",
"items": [
{
"type": "TextBlock",
"id": "alert-summary-title",
"text": "[{{- if gt (len .Alerts.Firing) 0 }}FIRING: {{ .Alerts.Firing | len }}] 🔥 {{end}} {{- if gt (len .Alerts.Resolved) 0 }}RESOLVED: {{ .Alerts.Resolved | len }}] ✅ {{end}} {{ with index .Alerts 0 -}}{{ .Labels.alertname }}{{ end }}",
"weight": "Bolder",
"color": "{{- if gt (len .Alerts.Firing) 0 }}Attention{{end}}{{- if gt (len .Alerts.Resolved) 0 }}Good{{end}}",
"size": "ExtraLarge",
"horizontalAlignment": "Left"
},
{
"type": "Container",
"id": "alert-summary-container",
"padding": "None",
"items": [
{
"type": "TextBlock",
"id": "alert-summary-description",
"text": "{{ .Alerts.Firing | len }} alerts are firing",
"wrap": true
},
{
"type": "ActionSet",
"id": "alert-silence-action",
"actions": [
{
"type": "Action.OpenUrl",
"id": "silence",
"title": "Silence",
"url": "{{ .ExternalURL }}/#/silences/new?filter=%7B
{{- range $key, $value := .CommonLabels }}
{{- if eq $key "alertname" }}{{ $key }}%3D%22{{ reReplaceAll "\\\\" "" $value }}%22{{ end }}
{{- end }}
{{- range $key, $value := .CommonLabels }}
{{- if ne $key "alertname" }}%2C{{ $key }}%3D%22{{ reReplaceAll "\\\\" "" $value }}%22{{ end }}
{{- end -}}%7D"
},
{{ with $alert := index .Alerts 0}}
{
"type": "Action.OpenUrl",
"id": "prom",
"title": "View in Prometheus",
"url": "{{ .GeneratorURL }}"
}
{{ end }}
]
}
]
},
{{ range $index, $alert := .Alerts.Firing }}
{
"type": "Container",
"id": "{{ $index }}-alerts-container",
"isVisible": true,
"padding": "None",
"items": [
{
"type": "Container",
"id": "alert-{{ $index }}-msg-container",
"padding": "None",
"separator": true,
"items": [
{
"type": "TextBlock",
"id": "alert-{{ $index }}-summary",
"text": "{{ $alert.Labels.alertname}}",
"wrap": true,
"size": "Medium",
"weight": "Bolder"
},
{
"type": "TextBlock",
"id": "alert-{{ $index }}-description",
"text": "{{ $alert.Annotations.description }}",
"wrap": true,
"weight": "Lighter",
"size": "Small"
},
{
"type": "ActionSet",
"id": "alert-{{ $index }}-actions",
"actions": [
{
"type": "Action.ShowCard",
"title": "View Labels",
"card": {
"type": "AdaptiveCard",
"body": [
{
"type": "TextBlock",
"wrap": true,
"text": "{{ $alert.Labels }}"
}
]
}
},
{
"type": "Action.ShowCard",
"title": "View Annotations",
"card": {
"type": "AdaptiveCard",
"body": [
{
"type": "TextBlock",
"wrap": true,
"text": "{{ $alert.Annotations }}"
}
]
}
}
]
},
{
"type": "Container",
"id": "alert-{{ $index }}-backup-labels-container",
"padding": "None",
"isVisible": false,
"items": [
{
"type": "FactSet",
"id": "alert-{{ $index }}-backup-labels-factset",
"facts": []
}
]
}
]
}
]
},
{{ end }}
{{ range $index, $alert := .Alerts.Resolved }}
{
"type": "Container",
"id": "{{ $index }}-alerts-container",
"isVisible": true,
"padding": "None",
"items": [
{
"type": "Container",
"id": "alert-{{ $index }}-msg-container",
"padding": "None",
"separator": true,
"items": [
{
"type": "TextBlock",
"id": "alert-{{ $index }}-summary",
"text": "{{ $alert.Labels.alertname}}",
"wrap": true,
"size": "Medium",
"weight": "Bolder"
},
{
"type": "TextBlock",
"id": "alert-{{ $index }}-description",
"text": "{{ $alert.Annotations.description }}",
"wrap": true,
"weight": "Lighter",
"size": "Small"
},
{
"type": "ActionSet",
"id": "alert-{{ $index }}-actions",
"actions": [
{
"type": "Action.ShowCard",
"title": "View Labels",
"card": {
"type": "AdaptiveCard",
"body": [
{
"type": "TextBlock",
"wrap": true,
"text": "{{ $alert.Labels }}"
}
]
}
},
{
"type": "Action.ShowCard",
"title": "View Annotations",
"card": {
"type": "AdaptiveCard",
"body": [
{
"type": "TextBlock",
"wrap": true,
"text": "{{ $alert.Annotations }}"
}
]
}
}
]
},
{
"type": "Container",
"id": "alert-{{ $index }}-backup-labels-container",
"padding": "None",
"isVisible": false,
"items": [
{
"type": "FactSet",
"id": "alert-{{ $index }}-backup-labels-factset",
"facts": []
}
]
}
]
}
]
},
{{ end }}
{
"type": "Container",
"id": "backup-alerts-container",
"isVisible": false,
"padding": "None",
"items": [
{
"type": "Container",
"id": "alert-catch-msg-container",
"padding": "None",
"separator": false,
"items": []
}
]
}
]
}
]
}
{{ end }}
receivers:
- name: sandbox
msteams_configs:
- webhook_url: "YOUR_WEBHOOK_HERE"
text: '{{ template "new.text" . }}'
Then I just send a POST with my JSON alert data to the Alertmanager /api/v2/alerts endpoint.
To me it seems like this one is not something that may fit everyones needs and indeed forming json via go templating is a portal to hell
For anyone looking for workaround you may do very simple trick:
From alertmanager side configure webhook receiver, aka:
receivers:
- name: default
webhook_configs:
- send_resolved: false
url: http://localhost:8080/demo
alertmanager will just send raw json payload to given url
and now you can code some simple, single endpoint service, that will take given input, transform it to whatever you wish to have in teams and send it to teams, aka:
Use designer to form message and samples for inspiration
Here is an example
const body = {
receiver: 'default',
status: 'firing',
alerts: [
{
status: 'firing',
labels: {
alertname: 'demo2',
component: 'bar',
severity: 'info',
},
annotations: {},
startsAt: '2024-05-13T06:11:03.188793747Z',
endsAt: '0001-01-01T00:00:00Z',
generatorURL: '',
fingerprint: 'c8002adfd87cf31c',
},
{
status: 'firing',
labels: {
alertname: 'demo2',
component: 'foo',
severity: 'info',
},
annotations: {},
startsAt: '2024-05-13T06:11:03.177669584Z',
endsAt: '0001-01-01T00:00:00Z',
generatorURL: '',
fingerprint: 'd31cdf353f33ac5b',
},
],
groupLabels: { alertname: 'demo2', severity: 'info' },
commonLabels: { alertname: 'demo2', severity: 'info' },
commonAnnotations: {},
externalURL: 'http://eb024ed65dcf:9093',
version: '4',
groupKey: '{}:{alertname="demo2", severity="info"}',
truncatedAlerts: 0,
}
const email =
body?.groupLabels?.annotation_owner ||
body?.commonLabels?.annotation_owner ||
body?.groupLabels?.owner ||
body?.commonLabels?.owner ||
body?.groupLabels?.tag_owner ||
body?.commonLabels?.tag_owner ||
body?.alerts?.[0]?.labels?.annotation_owner ||
body?.alerts?.[0]?.labels?.owner ||
body?.alerts?.[0]?.labels?.tag_owner ||
'[email protected]' // ''
const payload = {
type: 'message',
attachments: [
{
contentType: 'application/vnd.microsoft.card.adaptive',
content: {
$schema: 'http://adaptivecards.io/schemas/adaptive-card.json',
version: '1.0',
type: 'AdaptiveCard',
body: [
{
type: 'TextBlock',
weight: 'default',
text: `<at>${email}</at>`,
},
],
msteams: {
entities: [
{
type: 'mention',
text: `<at>${email}</at>`,
mentioned: {
id: email,
name: email,
},
},
],
},
},
},
],
}
for (const alert of body.alerts) {
const table = {
type: 'Table',
columns: [{ width: 1 }, { width: 1 }],
rows: [],
}
for (const [key, val] of Object.entries(alert.labels)) {
table.rows.push({
type: 'TableRow',
cells: [
{
type: 'TableCell',
items: [
{
type: 'TextBlock',
text: key,
wrap: true,
weight: 'default',
},
],
},
{
type: 'TableCell',
items: [
{
type: 'TextBlock',
text: val,
wrap: true,
weight: key === 'alertname' ? 'bolder' : 'default', // 'bolder'
color: 'default', // 'attention', 'good', 'warning' depending on key and val, aka status=firing - attention, severity=warning - warning
},
],
},
],
})
}
payload.attachments[0].content.body.push(table)
const actions = {
type: 'ActionSet',
actions: [],
}
if (alert.generatorURL) {
actions.actions.push({
type: 'Action.OpenUrl',
title: 'prometheus',
url: alert.generatorURL,
})
}
if (body.externalURL) {
actions.actions.push({
type: 'Action.OpenUrl',
title: 'alertmanager',
url: body.externalURL,
})
actions.actions.push({
type: 'Action.OpenUrl',
title: 'silence',
url: body.externalURL, // TODO: build silence link
})
}
actions.actions.push({
type: 'Action.OpenUrl',
title: 'readme',
url: `https://mac.atlassian.net/wiki/search?spaces=OPS&text=${alert.labels.alertname}`,
})
payload.attachments[0].content.body.push(actions)
}
console.log(JSON.stringify(payload, null, 4))
const res = await fetch(
'https://mac.webhook.office.com/webhookb2/.../IncomingWebhook/...',
{
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify(payload),
}
)
console.log(res.status) // 200
console.log(res.statusText) // 'OK'
console.log(await res.text()) // '1'
Note: js is used here only for example, if you wish you may go get alertmanager models directly, or even write everything as lua script in nginx
Breaking: Microsoft is deprecating "Connectors" in favor of "Power Automate Workflows": https://devblogs.microsoft.com/microsoft365dev/retirement-of-office-365-connectors-within-microsoft-teams/
Also a.f.a.i.k, the old "Message Card"-format is deprecated in this solution, and only "Adaptive Card" will work. So in order for AlertManager > Teams integration to work in August, this will need to be added.
Are there any concrete plans to add support for Adaptive Cards yet? If not I will start looking into the suggested workarounds in the linked issue.
