ol-infrastructure icon indicating copy to clipboard operation
ol-infrastructure copied to clipboard

Make Sentry spike alerts work for us rather than just being spam

Open feoh opened this issue 1 year ago • 0 comments

User Story

  • Right now we get quite a few Spike protection emails from Sentry. A quick scan indicates ballpark 20 a week.
  • This is too many. We can't meaningfully act on these and if any monitoring mechanism starts being too noisy it flips from increasing operational excellence to increasing operational burden.

Description/Context

We need to revisit our Sentry configuration. What do thse spike protection emails actually mean? Do we need to adjust the parameters Sentry uses for collection? Are there actual systemic problems that need to be dealt with in a concrete way?

Acceptance Criteria

  • [ ] Reduce the frequency of Sentry emails to the point where each one we get represents an action we should take. (That can include evaluating and determining no corrective action is required).
  • [ ] Create mechanisms for acting on Sentry E-mails because once the noise reduction is achieved, they should be actionable.

feoh avatar Apr 18 '23 14:04 feoh