Telegram polling discussion group fails

[Cenness]

What went wrong?

What happened:

I've installed OnCall via helm with telegram integration and polling enabled. Target channel has discussion enabled and bot is added to both.

In polling pod I see logs

/usr/local/lib/python3.12/site-packages/telegram/utils/request.py:49: UserWarning: python-telegram-bot is using upstream urllib3. This is allowed but not supported by python-telegram-bot maintainers.
  warnings.warn(
source=engine:app google_trace_id=none logger=engine.management.commands.start_telegram_polling Starting telegram polling...
source=engine:app google_trace_id=none logger=apscheduler.scheduler Scheduler started
source=engine:app google_trace_id=none logger=engine.management.commands.start_telegram_polling Update from Telegram: {'message': {'message_id': 28, 'forward_from_message_id': 2607, 'is_automatic_forward': True, 'supergroup_chat_created': False, 'delete_chat_photo': False, 'caption_entities': [], 'group_chat_created': False, 'forward_date': 1729153738, 'new_chat_members': [], 'forward_from_chat': {'id': -1009999999999, 'type': 'channel', 'title': 'Redacted alerts'}, 'text': 'alertname:  KubePersistentVolumeUsageCritical\n\nStatus: FIRING 🔥\n\nActive Alert List:\n\n================================\nLabel List:\n•     alertgroup = kubernetes-storage\n•     dev_team = AppDevOps\n•     env = dev\n•     instance = 10.10.1.10:10250\n•     metrics_path = /metrics\n•     namespace = monitoring\n•     node = worker-az0-8\n•     persistentvolumeclaim = oncall-test\n•     service = system\n•     severity = critical 🚨\n\nDescription:\n•• message = The persistent volume claimed by oncall-test in namespace monitoring has 4% free.\n•• timestamp = 2024-10-17 08:27:43.659 +0000 UTC\n\n\n Time Start Alert: 2024-10-17T08:28:13.561036624Z \n================================', 'forward_signature': 'Redacted Alert', 'sender_chat': {'id': -1009999999999, 'type': 'channel', 'title': 'Redacted alerts'}, 'new_chat_photo': [], 'channel_chat_created': False, 'photo': [], 'entities': [{'length': 10, 'type': 'bold', 'offset': 0}, {'length': 33, 'type': 'code', 'offset': 12}, {'length': 9, 'type': 'bold', 'offset': 55}, {'length': 18, 'type': 'bold', 'offset': 66}, {'length': 16, 'type': 'url', 'offset': 234}, {'length': 8, 'type': 'bot_command', 'offset': 272}, {'length': 50, 'type': 'italic', 'offset': 596}], 'chat': {'id': -1008888888888, 'type': 'supergroup', 'title': 'Redacted alerts Chat'}, 'date': 1729153741, 'from': {'first_name': 'Telegram', 'id': 777000, 'is_bot': False}}, 'update_id': 629379447}
source=engine:app google_trace_id=none logger=apps.telegram.updates.update_manager Processing update with handler: ChannelToGroupForwardHandler
source=engine:app google_trace_id=none logger=apps.telegram.updates.update_handlers.channel_to_group_forward Tried to send log and action message to comments, but organization deleted the channel connector. Channel chat id: -1009999999999. Channel message id: 2607. Group message id: 28.

ChatOps setting UI shows like there are no TELEGRAM.* env vars set:

but they are:

oncall-engine-7dd5dc9684-htdnc:/etc/app$ env | sort | grep -i teleg | grep -iv sd
FEATURE_TELEGRAM_INTEGRATION_ENABLED=True
FEATURE_TELEGRAM_LONG_POLLING_ENABLED=True
TELEGRAM_TOKEN=REDACTED:REDACTED
TELEGRAM_WEBHOOK_HOST=https://REDACTED

oncall-telegram-polling-7769f6d44d-rkfr5:/etc/app$ env | sort | grep -i teleg | grep -iv sd
FEATURE_TELEGRAM_INTEGRATION_ENABLED=True
FEATURE_TELEGRAM_LONG_POLLING_ENABLED=True
HOSTNAME=oncall-telegram-polling-7769f6d44d-rkfr5
TELEGRAM_TOKEN=REDACTED:REDACTED
TELEGRAM_WEBHOOK_HOST=https://REDACTED

oncall-celery-85695844b7-84p44:/etc/app$ env | sort | grep -i teleg | grep -iv sd
CELERY_WORKER_QUEUE=default,critical,long,slack,telegram,webhook,celery,grafana,retry
FEATURE_TELEGRAM_INTEGRATION_ENABLED=True
FEATURE_TELEGRAM_LONG_POLLING_ENABLED=True
TELEGRAM_TOKEN=REDACTED:REDACTED
TELEGRAM_WEBHOOK_HOST=https://REDACTED

What did you expect to happen:

chatops allowing to add my channels buttons appearing on alert messages

How do we reproduce it?

1. setup grafana oncall with telegram polling enabled
2. go to chatops tab - no options to add channel
3. go to polling logs - error "Tried to send log and action message to comments, but organization deleted the channel connector"

Grafana OnCall Version

v1.11.3

Product Area

Chatops

Grafana OnCall Platform?

Kubernetes

User's Browser?

Firefox 131

Anything else to add?

No response