generating backpressure dd events

[nirajdewani]

Description

Now that sentry supports Datadog events,
updating backpressure code to generate events as well (in addition to metrics).

Next steps

• Send events using dogshell
• Update event body if the need be

[lynnagara]

What is the motivation for this? Does it tell us anything that the metric doesn't already?

[mwarkentin]

Hey @lynnagara there was some more detail in the original PR / ticket: https://getsentry.atlassian.net/browse/OPS-5392

This would enable us to do things like:

• overlay events on top of dashboard widgets
• add tags to events to support analysis / grouping
• Trigger DD monitors for alerting purposes (eg. notifying slack / PagerDuty)

[mwarkentin]

Here's an example time period where we experienced backpressure on May 6:

And split:

Some things that aren't clear:

• Why is everything a multiple of 10?
• When exactly did backpressure trigger?
• When did it resolve?

Here is one ingest consumer lag graph of the same time:

With events we can have discrete events that could include information from the backpressure system for more context. These can be overlaid on top of graphs:

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 75.00000% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 77.89%. Comparing base (7eb565d) to head (51ae4cd). Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #71086      +/-   ##
==========================================
- Coverage   77.89%   77.89%   -0.01%     
==========================================
  Files        6525     6525              
  Lines      290597   290601       +4     
  Branches    50276    50276              
==========================================
- Hits       226375   226372       -3     
- Misses      57975    57979       +4     
- Partials     6247     6250       +3     

Files	Coverage Δ
src/sentry/processing/backpressure/health.py	83.78% <75.00%> (-0.51%)	:arrow_down:

... and 11 files with indirect coverage changes