getsentry
Trigger "The issue changes state from archived to escalating" not triggering alerts
Environment
SaaS (https://sentry.io/)
Steps to Reproduce
- Create an alert rule with the trigger: The issue changes state from archived to escalating
- Mark any issue as Archived until it happens 1 time(s)
- Send a new event to the issue above
- The issue is marked as escalating:
Escalated by Sentry because 1 event happened
Expected Result
The escalated issue trigger the alert rule
Actual Result
The alert rule is not triggered.
Alert rule settings: https://dev-curumas.sentry.io/issues/alerts/rules/notjava/15072297/details/?pageStatsPeriod=24h
Issue activity: https://dev-curumas.sentry.io/issues/6808289920/activity/?project=5242419&query=&referrer=issue-stream
Product Area
Alerts
Link
No response
DSN
No response
Version
No response
![linear[bot] avatar](https://avatars.githubusercontent.com/in/20150?v=4 "Published by a pub.dev verified publisher"){kind=small}
![getsantry[bot] avatar](https://avatars.githubusercontent.com/in/66573?v=4 "Published by a pub.dev verified publisher"){kind=small}
Hello, is this consistently reproducible like the trigger is not working at all for any alerts, or just sometimes? If sometimes, do we know which cases it's not working for? If you click the default "Archive until escalating" does that work?
@ceorourke Marking the issue as Archive until escalating option and then escalating it does trigger the alert.
Marking as archived until it affects a certain number of users or by time does not result in the alert triggering, but in both these cases the issue is marked as "Ongoing" instead of "Escalating". I don't expect the alert to trigger.
Hi, I created the original support ticket that led to this issue.
In my dashboard when I archive until it happens one more time and it does happen one more time, it is displayed as "Escalating".
What is the expected way to use sentry to get a notification the next time an error is triggered ?
I looked into this and the expected behavior is that this filter only triggers when you've selected the default "archive until escalating" option. However, if you have workflow notifications enabled you'll receive a notification for this without setting up an alert rule. Here's a screenshot from my test app I ran this in. I did the same setup where I archived the issue until it happened one more time from now, sent another event, and received this email:
The evaluation criteria for this filter is simple - we check if the event state has_escalated is True. In post process we only set has_escalated to True in 2 situations - if the issue was set to "archive until escalating" and a new event has come in (here) or if a new issue is escalating.
As a follow up I think we can add the notification I put a screenshot of to the workflow notification documentation and look into the issue state being marked as "escalating" in this scenario - in my opinion a single issue coming in isn't "escalating".
@ceorourke In that case, the UI should not set the issue as "Escalated" in the activity tab and "Escalating" in the issues feed after the "Archive until it happens once" action.
Because the UI is telling me "this issue went from archived to escalating" I expect the alert to trigger, but the alert did not detect it.
This inconsistency is the cause of this issue.
I've made a ticket for our backlog to address the issue substatus in this instance, thanks for the report!
Hey @ceorourke, A customer ran into the same problem. The WHEN trigger looks for "Archived to Escalating" However this is only triggered if Sentry's Escalating Algoritm is firing. Under the hood I found 2 flags: has_reappeared & has_escalated The first one is changed in the case when you manually set a condition like when 10 users or 1 single event. But the has_escalated flag only is changed when Sentry uses it's ML/forecasting to predict escalation.
The alert at this moment is not triggering because has_escalated is still false.
I think we need to improve this behaviour and also update the has_escalated flag to true if there are manual rules the users wants to escalate this from
Btw, I thought a little on this: From a user perspective they think the Archive until x (every option outside of escalating) means that once this happens it's escalating in their perception. But the main trigger is the is_escalated flag which is set by Sentry and not the user/manual. So maybe the ask here is to have an alert trigger that looks for has_reappeared or issues will be manually escalating but that seems more of a breaking change. TLDR: I think it's working as expected but the perception of the customers are different + there is no replacement to get what the customer wants. (getting alerts when their manual archived until X is triggered)
As mentioned above, we have a ticket in the backlog to address this confusion with archiving, thanks for looking into it!
Just re-iterating for all that we're tracking a backlog item to change how the UI sets the "issue as "Escalated" in the activity tab and "Escalating" in the issues feed after the "Archive until it happens once" action". It's possible this should be "ongoing" instead we still need to make that determination however.
Please note it is the intended behaviour that that "archive until happens x times" does not modify the issue state to be has_escalated.
It's worth mentioning that this seems to be a regression.
Our alerts used to be set up long before Sentry added the "archive until escalated" option. And they used to fire when issues were breaching the set up threshold. However, after certain update they all stopped.
It's also worth mentioning that sometimes setting a strict threshold over a number of events or affected users is preferred to non-determenistic "until escalated" approach. This gives monitoring people more flexibility and allow them to react quicker if things go completely wrong. Thus being able to setup an alert for this type of transition is essential.
Given all this, can it be prioritized higher?
@souredoutlook another customer reported the issue via Slack.
In addition to what Lingvistion mentioned, users expect the transition from "Archived" to "Escalated" to trigger an alert. Also, if this is "working as expected" there is no alert option to trigger an alert when an issue goes from "Archived" to "Ongoing". It makes the option to "archive until it happens X times" not very useful.
Can we get a re-evaluation on the "expected" behaviour here?
Closing the loop on this thread and some investigation:
- when we moved from "ignored" to "archived"
- we changed the issue alert trigger from "ignored to unresolved" to "archived to escalating" (see docs here: https://web.archive.org/web/20210610170040/https://docs.sentry.io/product/alerts-notifications/issue-alerts/#triggers)
- this left a gap where some cases that would have alerted in the past cannot be alerted on now - specifically because of the expected behavior that Colleen has outlined above.
- we're going to look into changing anything that is currently marked as "escalating" in the UI to be treated as escalated for the purposes of alerting to close this gap
Note: this will result in more alerts firing than in the past 2.5 years