‘Number of Errors’ Alert in Critical state for over a year

[kpujjigit]

Environment

SaaS (https://sentry.io/)

Steps to Reproduce

It appears this Number of Errors Alert

Expected Result

State of Alert should change, and we in-fact see a change in state when we go to Edit the rule:

Actual Result

Alert has been in a critical state since it was created a year ago; insight was lost for a client and spikes occurred without alerting to proper channels.

Product Area

Alerts

Link

No response

DSN

No response

Version

No response

[getsantry[bot]]

Auto-routing to @getsentry/product-owners-alerts for triage ⏲️

[schew2381]

@kpujjigit Hey there, could you link to the org and alert links somewhere? I don't see them on the issue rn

[leedongwei]

@kpujjigit We are still missing the link to org/alert for us to debug this.

[schew2381]

I was able to find the alert with a query. From looking at logs, the alert itself was setup in a way that it could never resolve.

The resolve threshold is set to resolve when # of errors is 100% lower than 1 week ago, but this means the number of errors at the current moment must essentially reach 0 in order resolve.

For example, we would reach this number very close to 100% if 1 week ago we had 9991 errors, and in the current moment we have 1

The alert was resolved by editing it, but it still has the same resolve issue so once it triggers it will never resolve as well.