Unexpected Failure Rate Alert Trigger
Environment
SaaS (https://sentry.io/)
What are you trying to accomplish?
An alert to check if Failure Rate exceeds 5% across a 2 hour interval. 5% denotes a warning state, and 7% denotes a critical state.
How are you getting stuck?
Alerts are firing when not supposed to. From the metric, we only see failure rate 3.77%, it is under 5%.
Where in the product are you?
Alerts
Link
No response
DSN
No response
Version
No response
Auto-routing to @getsentry/product-owners-alerts for triage ⏲️
@kpujjigit - thanks for filing! Is there a jira issue or place where we can get a link to the alert rule itself (realize that that's sensitive information and should not be captured here)
Hi all, client has been waiting for insight on this for over a month; any progress/update?
Hi @kpujjigit, do you have the Jira ticket where we can access this alert rule?
@mifu67 Here ya go: https://getsentry.atlassian.net/browse/FEEDBACK-2296
I'm looking into this, I might add some logging to figure out what's going on.
Hi @ceorourke Thanks for implementing logging, did you identify any potential root cause?
@kpujjigit I added logging but the issue hasn't happened again yet - if/when it does we should have some more insight.
Hello @kpujjigit apologizes it's been quite a while - I believe this is a duplicate of https://github.com/getsentry/sentry/issues/48456, see this comment specifically. I see the alert in question hasn't triggered in months so I can't say for sure (and the logging I'd added so long ago isn't there anymore) but it seems like it's just a chart bug rather than an actual triggering bug.