sentry icon indicating copy to clipboard operation
sentry copied to clipboard

Metric alert is sending duplicate notifications

Open cobyeastwood183 opened this issue 1 month ago • 5 comments

Environment

SaaS (https://sentry.io/)

Steps to Reproduce

  • configure a metric alert (see alert rules in linear)
  • duplicate it
  • delete one

Expected Result

  • receive one (slack) notification

Actual Result

  • receive two (slack) notifications

confirmed second alert was deleted, in this case it was deprecated. same alert id exists for both slack notifications while having different notification ids, details in linear.

Product Area

Alerts

Link

No response

DSN

No response

Version

No response

cobyeastwood183 avatar Dec 06 '25 00:12 cobyeastwood183

ENG-6063

linear[bot] avatar Dec 06 '25 00:12 linear[bot]

https://sentry.zendesk.com/agent/tickets/168670

cobyeastwood183 avatar Dec 06 '25 00:12 cobyeastwood183

Auto-routing to @getsentry/product-owners-alerts for triage ⏲️

getsantry[bot] avatar Dec 06 '25 00:12 getsantry[bot]

We weren't able to find any evidence of the deleted rule continuing to hang around and causing this, and we cannot reproduce the issue. We do not have records of the active alert firing multiple times, just the triggers and then a subsequent close ~5 minutes later. To double check since the provided screenshot doesn't show it, are the timestamps exactly the same when this happens? I can only see the timestamp on the first notification. I'm looking into enabling logs for tracking the double send here to get some more validation.

ceorourke avatar Dec 09 '25 00:12 ceorourke

We weren't able to find any evidence of the deleted rule continuing to hang around and causing this, and we cannot reproduce the issue. We do not have records of the active alert firing multiple times, just the triggers and then a subsequent close ~5 minutes later. To double check since the provided screenshot doesn't show it, are the timestamps exactly the same when this happens? I can only see the timestamp on the first notification. I'm looking into enabling logs for tracking the double send here to get some more validation.

Thanks for looking into this 🙌

These are the most recent timestamps 2025-12-08 08:01:07 am PST - https://pocketflows.sentry.io/alerts/rules/details/383607/?alert=162&referrer=metric_alert_slack&detection_type=static&notification_uuid=4f091379-9139-4049-8f98-6d2eec617f1d 2025-12-08 08:01:08 am PST - https://pocketflows.sentry.io/alerts/rules/details/383607/?alert=162&referrer=metric_alert_slack&detection_type=static&notification_uuid=489df0bb-badb-44e3-87b4-d67c5115a529

Got the links and the timestamp from our slack notifications. It was interesting to see that the resource url was the same alerts/rules/details/383607/?alert=162 but had two different notification_uuid

  • 4f091379-9139-4049-8f98-6d2eec617f1d
  • 489df0bb-badb-44e3-87b4-d67c5115a529

Please let me know if you'd like more examples

joemanley201 avatar Dec 09 '25 00:12 joemanley201

I have an additional case of this occurring where there are duplicate Slack notifications being sent from the same alert, confirmed that there are no overlapping Alerts causing this. Timestamps are identical, each duplicate notification has unique notification ID. See example duplicate alert notification urls along with alert links in internal ticket https://sentry.zendesk.com/agent/tickets/169513

ReneGreen27 avatar Dec 11 '25 21:12 ReneGreen27

Hey @cobyeastwood183 @joemanley201 @ReneGreen27, My guess is this might be a bug with our deduping logic for warning/critical thresholds with the same notification action. We're trying to repro and fix now.

mifu67 avatar Dec 11 '25 23:12 mifu67

Hey @cobyeastwood183 @joemanley201 @ReneGreen27, My guess is this might be a bug with our deduping logic for warning/critical thresholds with the same notification action. We're trying to repro and fix now.

thank you so much! just had another occurrence 9 minutes ago, so let me know if you need that information as well.

in our case:

  • we had a legacy alert and once we saw the warning about deprecation we created a new alert with the same params
  • deleted the legacy alert
  • saw duplicate notifications come in

joemanley201 avatar Dec 11 '25 23:12 joemanley201