fix(notify): safe telegram html truncation

[timmilesdw]

Fix telegram truncation breaking markup

I've come back 4 years later to fix the bug I introduced with telegram notifier in 2022.

Context:
When truncating telegram alert with HTML formatting, truncate function would cut off mid html tag, leaving malformed HTML. Added TruncateInRunesHTML function that fixes this bug. It tracks open tags while scanning and ensures that all the tags are properly closed. It finds the last safe cut point where we can fit the truncation marker plus all necesarry closing tags within the limit.

Tested with unit tests which I also added, and also manually with a long alert message, looks something like this:

🔥 TruncationTest 🔥

Status: firing
Severity: critical

Labels:
  • alertname: TruncationTest
  • instance: server-001.example.com:9090
  • job: prometheus
  • label_01: value_for_label_01_with_some_padding
  • label_02: value_for_label_02_with_some_padding
  • label_03: value_for_label_03_with_some_padding
  • label_04: value_for_label_04_with_some_padding
  • label_05: value_for_label_05_with_some_padding
  • label_06: value_for_label_06_with_some_padding
  • label_07: value_for_label_07_with_some_padding
  • label_08: value_for_label_08_with_some_padding
  • label_09: value_for_label_09_with_some_padding
  • label_10: value_for_label_10_with_some_padding
  • label_11: value_for_label_01_with_some_padding
  • label_12: value_for_label_02_with_some_padding
  • label_13: value_for_label_03_with_some_padding
  • label_14: value_for_label_04_with_some_padding
  • label_15: value_for_label_05_with_some_padding
  • label_16: value_for_label_06_with_some_padding
  • label_17: value_for_label_07_with_some_padding
  • label_18: value_for_label_08_with_some_padding
  • label_19: value_for_label_09_with_some_padding
  • label_20: value_for_label_10_with_some_padding
  • label_21: value_for_label_01_with_some_padding
  • label_22: value_for_label_02_with_some_padding
  • label_23: value_for_label_03_with_some_padding
  • label_24: value_for_label_04_with_some_padding
  • label_25: value_for_label_05_with_some_padding
  • label_26: value_for_label_06_with_some_padding
  • label_27: value_for_label_07_with_some_padding
  • label_28: value_for_label_08_with_some_padding
  • label_29: value_for_label_09_with_some_padding
  • label_30: value_for_label_10_with_some_padding
  • label_31: value_for_label_01_with_some_padding
  • label_32: value_for_label_02_with_some_padding
  • label_33: value_for_label_03_with_some_padding
  • label_34: value_for_label_04_with_some_padding
  • label_35: value_for_label_05_with_some_padding
  • label_36: value_for_label_06_with_some_padding
  • label_37: value_for_label_07_with_some_padding
  • label_38: value_for_label_08_with_some_padding
  • label_39: value_for_label_09_with_some_padding
  • label_40: value_for_label_10_with_some_padding
  • namespace: monitoring
  • pod: prometheus-server-0
  • service: prometheus-operated
  • severity: critical

Annotations:
  • description: This is a very long description that will be repeated many times to ensure we exceed the Telegram message limit of 4096 characters. This is a very long description that will be repeated many times to ensure we exceed the Telegram message limit of 4096 characters. This is a very long description that will be repeated many times to ensure we exceed the Telegram message limit of 4096 characters. This is a very long description that will be repeated many times to ensure we exceed the Telegram message limit of 4096 characters. This is a very long description that will be repeated many times to ensure we exceed the Telegram message limit of 4096 characters. This is a very long description that will be repeated many times to ensure we exceed the Telegram message limit of 4096 characters. This is a very long description that will be repeated many times to ensure we exceed the Telegram message limit of 4096 characters. This is a very long description that will be repeated many times to ensure we exceed the Telegram message limit of 4096 chara…

Closes related issue #2923

[timmilesdw]

Hello! @simonpasquier @SuperQ can i get a review on this pr?

[TheMeier]

I am not really shure it is a good idea. Is it HTLM sanitization really a functionality alertmanager should provide?

There is some alternatives.
You don't need to render all labels in the templates (limit it to 5 or so), you could go a route like described in https://docs.exordis.com/blog/telegram-notifications-from-alertmanager-with-respect-to-4096-chars-limit/, you can use an adapter/middleware that receives webhook notifications and passes them on to telegram (https://github.com/slok/alertgram).

[timmilesdw]

I am not really shure it is a good idea. Is it HTLM sanitization really a functionality alertmanager should provide?

There is some alternatives. You don't need to render all labels in the templates (limit it to 5 or so), you could go a route like described in https://docs.exordis.com/blog/telegram-notifications-from-alertmanager-with-respect-to-4096-chars-limit/, you can use an adapter/middleware that receives webhook notifications and passes them on to telegram (https://github.com/slok/alertgram).

Thanks for the feedback!

I understand your concerns, but this is just a pretty simple and robust truncation function. This PR is not adding general-purpose HTML sanitization to Alertmanager. It does not attempt to clean, filter, or validate arbitrary user-supplied HTML.

The alternatives you suggest are basically workarounds: limiting templates, using external adapters, or avoiding truncation entirely. Those can be useful, but they don’t address the underlying problem that Alertmanager itself produces invalid HTML when truncating. As a bonus, the same truncation function can be reused by other notifiers that produce HTML output (If there will be any).