Metric alert is sending duplicate notifications
Environment
SaaS (https://sentry.io/)
Steps to Reproduce
- configure a metric alert (see alert rules in linear)
- duplicate it
- delete one
Expected Result
- receive one (slack) notification
Actual Result
- receive two (slack) notifications
confirmed second alert was deleted, in this case it was deprecated. same alert id exists for both slack notifications while having different notification ids, details in linear.
Product Area
Alerts
Link
No response
DSN
No response
Version
No response
https://sentry.zendesk.com/agent/tickets/168670
Auto-routing to @getsentry/product-owners-alerts for triage ⏲️
We weren't able to find any evidence of the deleted rule continuing to hang around and causing this, and we cannot reproduce the issue. We do not have records of the active alert firing multiple times, just the triggers and then a subsequent close ~5 minutes later. To double check since the provided screenshot doesn't show it, are the timestamps exactly the same when this happens? I can only see the timestamp on the first notification. I'm looking into enabling logs for tracking the double send here to get some more validation.
We weren't able to find any evidence of the deleted rule continuing to hang around and causing this, and we cannot reproduce the issue. We do not have records of the active alert firing multiple times, just the triggers and then a subsequent close ~5 minutes later. To double check since the provided screenshot doesn't show it, are the timestamps exactly the same when this happens? I can only see the timestamp on the first notification. I'm looking into enabling logs for tracking the double send here to get some more validation.
Thanks for looking into this 🙌
These are the most recent timestamps 2025-12-08 08:01:07 am PST - https://pocketflows.sentry.io/alerts/rules/details/383607/?alert=162&referrer=metric_alert_slack&detection_type=static¬ification_uuid=4f091379-9139-4049-8f98-6d2eec617f1d 2025-12-08 08:01:08 am PST - https://pocketflows.sentry.io/alerts/rules/details/383607/?alert=162&referrer=metric_alert_slack&detection_type=static¬ification_uuid=489df0bb-badb-44e3-87b4-d67c5115a529
Got the links and the timestamp from our slack notifications. It was interesting to see that the resource url was the same alerts/rules/details/383607/?alert=162 but had two different notification_uuid
- 4f091379-9139-4049-8f98-6d2eec617f1d
- 489df0bb-badb-44e3-87b4-d67c5115a529
Please let me know if you'd like more examples
I have an additional case of this occurring where there are duplicate Slack notifications being sent from the same alert, confirmed that there are no overlapping Alerts causing this. Timestamps are identical, each duplicate notification has unique notification ID. See example duplicate alert notification urls along with alert links in internal ticket https://sentry.zendesk.com/agent/tickets/169513
Hey @cobyeastwood183 @joemanley201 @ReneGreen27, My guess is this might be a bug with our deduping logic for warning/critical thresholds with the same notification action. We're trying to repro and fix now.
Hey @cobyeastwood183 @joemanley201 @ReneGreen27, My guess is this might be a bug with our deduping logic for warning/critical thresholds with the same notification action. We're trying to repro and fix now.
thank you so much! just had another occurrence 9 minutes ago, so let me know if you need that information as well.
in our case:
- we had a legacy alert and once we saw the warning about deprecation we created a new alert with the same params
- deleted the legacy alert
- saw duplicate notifications come in