Truncating Telegram HTML/Markdown alerts returns 400

[fopina]

Up to recently I thought that Grafana did not truncate messages to 4096 and that was the reason I kept getting errors 400 from Telegram API (in the logs).

When considering doing a PR I found https://github.com/grafana/alerting/blob/main/receivers/telegram/telegram.go#L120 and tested simpler alerts and indeed it was truncated and alert sent successfully.

Then I found somewhere that Telegram requires messages with Markdown/HTML parse_mode to be valid and I finally managed to find my issue: truncating often removes a terminating tag (as I use HTML) leaving the opening tag, resulting in invalid HTML.

I've reproduced this easily using Contact points test message. I just filled a message <b>a</b> preceeded by 4089 A and clicking Test results in error 400.

Error 400s are easily confirmed by just using <b>a as message or _a with Markdown parse mode.

I usually like to provide possible solutions yet I don't have an easy one in this case 😞

I believe the proper way would be to, after truncating, to loosely parse the message (in the proper parse_mode) and close any blocks that are missing, yet "sounds heavy"...

A simpler workaround could be to a new feature which is to use allow a Contact Point to have "fallback" integrations. Currently one with multiple integrations will always notify all integrations. If there was a "fallback" integration, it would mean it's only notified if a default one fails. This would allow to set up a contact point with 2 Telegram integrations: one with pretty markdown/html template and a second with a simpler template and parse_mode None.