ref(backpressure): Track reasons for unhealthy consumers and services

[loewenheim]

trafficstars

This refactors backpressure management so that when a service exceeds its memory threshold or there's an exception when checking, that information is attached to the Sentry error for that service. Likewise, when a consumer is unhealthy, the error contains the list of unhealthy services.

This doesn't change the logic of backpressure management in any way, only the presentation. Unless I've introduced a bug somewhere, which I would obviously never do.

[codecov[bot]]

Codecov Report

Attention: 11 lines in your changes are missing coverage. Please review.

Comparison is base (70898e1) 81.37% compared to head (f7ffda9) 81.39%. Report is 50 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #64998      +/-   ##
==========================================
+ Coverage   81.37%   81.39%   +0.01%     
==========================================
  Files        5242     5251       +9     
  Lines      231348   232102     +754     
  Branches    45384    45573     +189     
==========================================
+ Hits       188266   188909     +643     
- Misses      37230    37303      +73     
- Partials     5852     5890      +38     

Files	Coverage Δ
src/sentry/processing/backpressure/memory.py	75.67% <87.50%> (+1.38%)	:arrow_up:
src/sentry/processing/backpressure/monitor.py	61.11% <76.92%> (+0.54%)	:arrow_up:
src/sentry/processing/backpressure/health.py	84.05% <65.00%> (-1.91%)	:arrow_down:

... and 60 files with indirect coverage changes

[sentry[bot]]

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

• ‼️ KeyError: 'total_system_memory' backpressure.monitoring View Issue

_{Did you find this useful? React with a 👍 or 👎}

[getsentry-bot]

PR reverted: 956db53336cc15a019ea84aad6948a4d390673ce

[ykamo001]

I think reverting this might have caused new monitor failures: https://sentry.sentry.io/issues/?project=1&query=is%3Aunresolved&referrer=issue-list&sort=date&statsPeriod=1h