sentry icon indicating copy to clipboard operation
sentry copied to clipboard

Use Sentry issue priority to specify `severity` on outgoing events

Open tiny-crab opened this issue 1 year ago • 8 comments

Problem Statement

My team uses Sentry to create incidents on Pagerduty when a new event is added to an issue. Sometimes there are noisy issues that come from error logs or thrown exceptions, and we want to be able to control the noise without outright archiving the issue.

I set up Event Orchestration in Pagerduty that sets the urgency of the incident based on the severity sent by Sentry. When the severity is error, we page the on-call. When the severity is warning, we file an incident but do not page the oncall.

I'd like to be able to set a severity level based on the Sentry Issue priority, rather than the underlying event level. This way, if a noisy issue pages the on-call, they would be able to set the Issue Priority in Sentry to Medium. Then they will still receive incidents, but won't be woken up in the middle of the night from a page.

Image

Sentry currently only allows me to filter based off of event level, not Issue Priority.

Image

Solution Brainstorm

I'd like to be able to set a filter based off of Issue Priority. This way, I could set up an alert that declares:

  1. When an event is captured by Sentry, and the Issue's Priority is High
  2. Send a notification to Pagerduty with Error severity

And another alert that could declare:

  1. When an event is captured by Sentry, and the Issue's Priority is Medium
  2. Send a notification to Pagerduty with Warning severity.

And so on for Low priority issues.

Product Area

Alerts

tiny-crab avatar Sep 23 '24 22:09 tiny-crab

Assigning to @getsentry/support for routing ⏲️

getsantry[bot] avatar Sep 23 '24 22:09 getsantry[bot]

Routing to @getsentry/product-owners-alerts for triage ⏲️

getsantry[bot] avatar Sep 25 '24 10:09 getsantry[bot]

@tiny-crab - thank you for filing this feature idea! We're making some bigger changes to how alerts are configured, and filtering based on priority is on our list of changes to add in

rachrwang avatar Sep 27 '24 19:09 rachrwang

@rachrwang That's great to hear. I'm looking forward to seeing the new features. What's the timeline for this?

tiny-crab avatar Sep 30 '24 21:09 tiny-crab

Hey @tiny-crab, I think as a very rough estimate I'd say EOY or beginning of next year.

schew2381 avatar Oct 01 '24 20:10 schew2381

Okay, that's good to hear. Excited for that to be built and released. How will I know when it's complete?

tiny-crab avatar Oct 01 '24 20:10 tiny-crab

We have some larger UI changes planned, so it should be very noticeable when it takes effect 😅

schew2381 avatar Oct 02 '24 22:10 schew2381

Cool, looking forward to seeing the new changes.

tiny-crab avatar Oct 03 '24 16:10 tiny-crab

Just checking in, is this feature still likely to be added by EOY? I'm scoping out some work to work around this in the meantime, but curious how your roadmap looks in the next couple of months.

tiny-crab avatar Nov 12 '24 16:11 tiny-crab

@tiny-crab - our changes will start rolling out in the first half of next year. Thanks!

rachrwang avatar Nov 12 '24 17:11 rachrwang

@tiny-crab we have added these new priority filters that might meet your needs:

Image

ceorourke avatar Dec 12 '24 21:12 ceorourke

@ceorourke I just checked those out, thanks for the tip.

What does it mean by marks in those two filters? Does the priority need to change in order to trigger an alert using that event type?

If I'm not mistaken, I'm more looking for a filter in the IF section of the alert configuration. Since I want to manually change the Sentry issue priority based on noise, and have it sent at a lower severity to Pagerduty.

tiny-crab avatar Dec 30 '24 21:12 tiny-crab

If I'm not mistaken, I'm more looking for a filter in the IF section of the alert configuration. Since I want to manually change the Sentry issue priority based on noise, and have it sent at a lower severity to PagerDuty.

Yeah, you're correct that you'd want to have the if section create a filter to then send that to pager duty. The marks info is basically saying "If an issue's priority is set or changed to high, then trigger the following actions"


As @rachrwang mentioned, we're actively working on a feature to make this waayyy more robust. We'll have a configuration to allow you to define the issue priority as you described. That way you could just create issues that are lower priority for your team as low priority issues. We'll also introduce the if filter that you described as a way to handle the severity levels. We think the combination of features will allow you to really hone into when / how to be alerted about issues.

saponifi3d avatar Jan 02 '25 21:01 saponifi3d

@saponifi3d Thanks! That's exciting to hear. @rachrwang had mentioned these features would be rolling out in H1 2025... any updates on how soon that would be? :D

tiny-crab avatar Jan 21 '25 20:01 tiny-crab

@tiny-crab - we don't have any updates yet on timing, though we will definitely follow up as we have more details to share!

rachrwang avatar Jan 23 '25 19:01 rachrwang

Just checking in to see if there’s any update on this @rachrwang

Shivang-Bhandari avatar Feb 18 '25 13:02 Shivang-Bhandari

We're still actively working on this, it should be released in the new few months but I can't make promises regarding the timeline.

ceorourke avatar Feb 18 '25 19:02 ceorourke