sentry icon indicating copy to clipboard operation
sentry copied to clipboard

Unexpected Failure Rate Alert Trigger

Open kpujjigit opened this issue 1 year ago • 8 comments

Environment

SaaS (https://sentry.io/)

What are you trying to accomplish?

An alert to check if Failure Rate exceeds 5% across a 2 hour interval. 5% denotes a warning state, and 7% denotes a critical state.

How are you getting stuck?

Alerts are firing when not supposed to. From the metric, we only see failure rate 3.77%, it is under 5%.

Image

Where in the product are you?

Alerts

Link

No response

DSN

No response

Version

No response

kpujjigit avatar Aug 15 '24 20:08 kpujjigit

Auto-routing to @getsentry/product-owners-alerts for triage ⏲️

getsantry[bot] avatar Aug 15 '24 20:08 getsantry[bot]

@kpujjigit - thanks for filing! Is there a jira issue or place where we can get a link to the alert rule itself (realize that that's sensitive information and should not be captured here)

rachrwang avatar Aug 15 '24 21:08 rachrwang

Hi all, client has been waiting for insight on this for over a month; any progress/update?

kpujjigit avatar Sep 12 '24 20:09 kpujjigit

Hi @kpujjigit, do you have the Jira ticket where we can access this alert rule?

mifu67 avatar Sep 12 '24 21:09 mifu67

@mifu67 Here ya go: https://getsentry.atlassian.net/browse/FEEDBACK-2296

kpujjigit avatar Sep 17 '24 20:09 kpujjigit

I'm looking into this, I might add some logging to figure out what's going on.

ceorourke avatar Sep 17 '24 23:09 ceorourke

Hi @ceorourke Thanks for implementing logging, did you identify any potential root cause?

kpujjigit avatar Oct 02 '24 20:10 kpujjigit

@kpujjigit I added logging but the issue hasn't happened again yet - if/when it does we should have some more insight.

ceorourke avatar Oct 02 '24 20:10 ceorourke

Hello @kpujjigit apologizes it's been quite a while - I believe this is a duplicate of https://github.com/getsentry/sentry/issues/48456, see this comment specifically. I see the alert in question hasn't triggered in months so I can't say for sure (and the logging I'd added so long ago isn't there anymore) but it seems like it's just a chart bug rather than an actual triggering bug.

ceorourke avatar Sep 30 '25 23:09 ceorourke