alertmanager Telegram can't parse entities after message truncation with parse

What did you do? Used Telegram integration with parse_mode="HTML"

What did you expect to see? Alert notification in Telegram chat

What did you see instead? Under which circumstances? Alert notification wasn't sent to chat

Environment

System information:

Linux 5.4.0-109-generic x86_64
Alertmanager version:

alertmanager, version 0.24.0 (branch: HEAD, revision: f484b17fa3c583ed1b2c8bbcec20ba1db2aa5f11) build user: root@265f14f5c6fc build date: 20220325-09:31:33 go version: go1.17.8 platform: linux/amd64
Prometheus version:

prometheus, version 2.34.0 (branch: HEAD, revision: 881111fec4332c33094a6fb2680c71fffc427275) build user: root@121ad7ea5487 build date: 20220315-15:18:00 go version: go1.17.8 platform: linux/amd64
Alertmanager configuration file:

global:
  resolve_timeout: 3m
templates:
- '/etc/alertmanager/templates/*.tmpl'
receivers:
- name: alerts-test1
  telegram_configs:
  - api_url: https://api.telegram.org
    bot_token: 1657974151:BBBO9Iggg_Tif04Tg9asdfSuR83niyNOSHE  # has been changed for security reasons
    chat_id: -10075044612354    # has been changed for security reasons
    disable_notifications: false
    message: '{{ template "telegram.html.message" . }}'
    parse_mode: HTML
    send_resolved: true

route:
  group_by:
  - alertname
  group_interval: 30s
  group_wait: 15s
  receiver: alerts-test1
  repeat_interval: 1d

telegram.html.message template:

{{ define "telegram.html.message" }}
{{ range .Alerts }}
{{ if eq .Status "firing"}}🔥 <b>{{ .Labels.alertname }}</b> 🔥{{ else }}✅ <b>{{ .Labels.alertname }}</b> ✅{{ end }}
<b>Labels:</b>{{ range $key, $value := .Labels }}{{ if ne $key "alertname" }}
    <b>{{ $key }}</b>: <i>{{ $value }}</i>{{ end }}{{ end }}
<b>Annotations:</b>{{ range $key, $value := .Annotations }}
    <b>{{ $key }}</b>: <i>{{ $value }}</i>{{ end }}
{{ end }}
{{ end }}

Prometheus configuration file: Isn't relevant to the issue
Logs:

May 16 11:21:59 focal-1 alertmanager[30906]: ts=2022-05-16T08:21:59.430Z caller=telegram.go:72 level=debug integration=telegram msg="truncated message" truncated_message="\n\n🔥 <b>ExporterDown</b> 🔥\n
<b>Labels:</b>\n    <b>address</b>: <i>192.168.56.10</i>\n    <b>environment</b>: <i>testing</i>\n    <b>instance</b>: <i>192.168.56.10:9100</i>\n    <b>job</b>: <i>node_exporter</i>\n    <b>label10</b>
: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label11</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label12</b>: <i>valuevaluevaluevaluevaluevaluevaluevalu
evaluevalue</i>\n    <b>label13</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label2</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label3</b>: <i>valuev
aluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label4</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label5</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</
i>\n    <b>label6</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label7</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label8</b>: <i>valuevaluevaluevalue
valuevaluevaluevaluevaluevalue</i>\n    <b>label9</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>severity</b>: <i>critical</i>\n<b>Annotations:</b>\n    <b>description</b>: <i>Ca
n't get data from exporter</i>\n    <b>space_filler</b>: <i>Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Te
st Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test</i>\n    <b>summary</b>:
 <i>Exporter down</i>\n\n🔥 <b>ExporterDown</b> 🔥\n<b>Labels:</b>\n    <b>address</b>: <i>192.168.56.20</i>\n    <b>environment</b>: <i>testing</i>\n    <b>instance</b>: <i>192.168.56.20:9100</i>\n    
<b>job</b>: <i>node_exporter</i>\n    <b>label10</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label11</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>lab
el12</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label13</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label2</b>: <i>valuevaluevaluevaluevaluevalueva
luevaluevaluevalue</i>\n    <b>label3</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label4</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label5</b>: <i>
valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label6</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label7</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluev
alue</i>\n    <b>label8</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label9</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>severity</b>: <i>critical</i>
\n<b>Annotations:</b>\n    <b>description</b>: <i>Can't get data from exporter</i>\n    <b>space_filler</b>: <i>Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test 
Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Test Te
st Test Test Test Test Test</i>\n    <b>summary</b>: <i>Exporter down</i>\n\n🔥 <b>ExporterDown</b> 🔥\n<b>Labels:</b>\n    <b>address</b>: <i>192.168.56.30</i>\n    <b>environment</b>: <i>testing</i>\n
    <b>instance</b>: <i>192.168.56.30:9100</i>\n    <b>job</b>: <i>node_exporter</i>\n    <b>label10</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label11</b>: <i>valuevaluevalu
evaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label12</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label13</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n  
  <b>label2</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label3</b>: <i>valuevaluevaluevaluevaluevaluevaluevaluevaluevalue</i>\n    <b>label4</b>: <i>valuevaluevaluevaluevaluev
aluevaluevaluevaluevalue</i>\n    <b>label5</b>: <i>valuevaluevaluevaluevaluevaluevaluevalu..."

May 16 11:21:59 focal-1 alertmanager[30906]: ts=2022-05-16T08:21:59.526Z caller=notify.go:732 level=warn component=dispatcher receiver=alerts-test1 integration=telegram[0] msg="Notify attempt failed, wi
ll retry later" attempts=1 err="telegram: Bad Request: can't parse entities: Can't find end tag corresponding to start tag i (400)"

May 16 '22 08:05 iuhenio

Hi, having same issue here. Did anyone find a workaround or was able to solve this? @metalmatze @timmilesdw

Sep 26 '22 08:09 petshopjke1337

Hello. I had the same problem. Try changing the parsing method to Markdown.

https://prometheus.io/docs/alerting/latest/configuration/#telegram_config

Change "<b>" and "</b>" in the template to ** Change "<i>" and "</i>" to *

Oct 19 '22 11:10 compudaster

having same issue here, need help

May 17 '23 08:05 1014670860

Hello. I had the same problem. Try changing the parsing method to Markdown.

https://prometheus.io/docs/alerting/latest/configuration/#telegram_config

Change "" and "" in the template to ** Change "" and "" to * It's not really an option for us. Markdown is buggy and lack some functions. It's been over a year, i guess telegram integration is not that popular...

May 19 '23 21:05 petshopjke1337

I used this workaround until we have a better solution:

        {{- $alertsLen := (len .Alerts) -}}
        {{ range .Alerts }}

            {{- if .Annotations.description }}
    _{{ .Annotations.description }}_
            {{- end }}
            {{- if .Annotations.message }}
    _{{ .Annotations.message }}_
            {{- end }}
            {{- if le $alertsLen 10 }}
    [Prometheus]({{ .GeneratorURL }})
            {{- end }}
        {{ end }}
    {{- end }}

Aug 17 '23 06:08 pando85

I am experiencing the same issue. This is not related directly to HTML but rather to Telegram's message size limitation. Once you reach the max size of the message's length, the message is truncated by alertmanager, see:

https://github.com/prometheus/alertmanager/blob/14cbe6301c732658d6fe877ec55ad5b738abcf06/notify/telegram/telegram.go#L82-L87

Now this might cause an HTML tag or Markdown symbol to be sent unclosed which will cause this error.

Honestly, I'm not sure how this can be fixed properly, it might be better to catch this error and react with a stripped text or even sending an .html/.md file to at least make sure that the alert arrives to its destination.

EDIT:

It might be possible to add another configuration next to group_by, like max_alerts_in_group and allow_bulk which will make sure that each message contains its alerts and N messages are sent when they pass Telegram's limitation.

Mar 29 '24 15:03 Shaked

can be configured so it send a message per alert?

Jul 15 '24 22:07 segator

still having the same issue, the max_alerts_in_group and allow_bulk can pass the telegram limitation, adding theses options will save us

Aug 08 '24 19:08 mhdan

+1 :+1: Still highly relevant. Especially when using cAdvisor, whose alerts contain a large number of labels, so the message is truncated and it breaks HTML structure

Dec 10 '24 15:12 victorryakh

Just ran into the same issue and developed a template for telegram notifications with following logic

build full notification message with all alerts
if it is too big fall back to less verbose single alert format (no labels)
if even with less verbose format the message appears too big - remove some alerts to fit the size limit (alerts are removed as a whole so telegram does not complain about broken markdown) and add notice about truncation to the message

I use MarkdownV2 as parse_mode, but the same approach may be used for HTML

{{- define "telegram.alert_list" -}}
  {{- $maxLength := 3900 -}}
  {{- $message := "" -}}
  {{- $message_is_truncated := false -}}
  {{- $message_no_labels := "" -}}
  {{- $message_no_labels_is_truncated := false -}}

  {{- range .Alerts -}}
    {{- $escape_regex := "([-_*\\[\\]()~`>#+=|{}.!])" -}}
    
    {{- /* Build alert text */ -}}
    {{- $alertText := ""}}
    {{- if eq .Status "firing" }}
      {{- $alertText = "🔥 "}}
    {{- end -}}
    {{- if eq .Status "resolved" }}
      {{- $alertText = "✅ "}}
    {{- end -}}

    {{- $title := .Labels.alertname | reReplaceAll  $escape_regex "\\$1" -}}
    {{- if .Annotations.summary}}
      {{- $title = printf "%s \\- %s" $title (.Annotations.summary | reReplaceAll  $escape_regex "\\$1")  -}}
    {{- end }}

    {{- $alertText = printf "%s***\\<%s\\> %s***\n" $alertText ( or .Labels.severity "no severity") $title -}}
    {{- if .Annotations.description}}
      {{- $alertText = printf "%s_%s_\n" $alertText ( .Annotations.description | reReplaceAll $escape_regex "\\$1" ) -}}
    {{- end }}
    
    {{- if .GeneratorURL -}}
      {{- $alertText = printf "%s[Source](%s)" $alertText (.GeneratorURL) -}}
    {{- end }}

    {{- if .Annotations.runbook_url -}}
      {{- $alertText = printf "%s [Runbook](%s)" $alertText (.Annotations.runbook_url) -}}
    {{- end -}}

    {{- /* Store alert text without labels in case all alerts would not fit message length limit  */ -}}
    {{- $message_tmp := "" -}}
    {{- if $message_no_labels  -}}
      {{- $message_tmp = printf "%s\n\n%s" $message_no_labels $alertText -}}
    {{- else -}}
      {{- $message_tmp = printf "%s" $alertText -}}
    {{- end -}} 

    {{- if $message_tmp | len | gt $maxLength  -}}
      {{- $message_no_labels = $message_tmp -}}
    {{- else -}}
      {{- $message_no_labels_is_truncated = true -}}
    {{- end -}}

    {{- /* Add labels to alert text */ -}}
    {{- range .Labels.SortedPairs -}}
      {{- if (eq .Name "alertname" "description" "summary" "runbook_url" "severity" "prometheus" ) | not -}}
      {{- $alertText = printf "%s\n🏷 %s: `%s`" $alertText (.Name | reReplaceAll $escape_regex "\\$1" ) .Value -}}
      {{- end -}}
    {{- end }}

    {{- $message_tmp := "" -}}
    {{- if $message  -}}
      {{- $message_tmp = printf "%s\n\n%s" $message $alertText -}}
    {{- else -}}
      {{- $message_tmp = printf "%s" $alertText -}}
    {{- end -}}
    
    {{- if $message_tmp | len | gt $maxLength  -}}
      {{- $message = $message_tmp -}}
    {{- else -}}
      {{- $message_is_truncated = true -}}
    {{- end -}}
    
  {{- end -}}

  {{- $message =  printf "%s\n" $message -}}
  {{- if $message_is_truncated -}}
    {{- $message =  printf "%s\nMessage is truncated to fit telegram API limit" $message -}}
  {{- end -}}

  {{- $message_no_labels =  printf "%s\n" $message_no_labels -}}
  {{- if $message_no_labels_is_truncated -}}
    {{- $message_no_labels =  printf "%s\nMessage is truncated to fit telegram API limit" $message_no_labels -}}
  {{- end -}}

  {{- if $message_is_truncated -}}
    {{- $message_no_labels -}}
  {{- else -}}
    {{- $message -}}
  {{- end -}}
{{- end -}}

{{- define "telegram.message" -}}
🔥  firing: {{ len .Alerts.Firing }} ✅ resolved: {{ len .Alerts.Resolved }}

{{ template "telegram.alert_list" . }}
[See On AlertManager]({{ .ExternalURL }}/#/alerts?receiver={{ .Receiver }})
\#alerts
{{- end -}}

Here is how it looks in telegram:

UPD: wrote a blog post with template updated to output firing alerts first and more inline comments

Dec 15 '24 14:12 KonstantinNosov

Telegram can't parse entities after message truncation with parse_mode="HTML"