feat(hybridcloud) Add tasks for webhookpayload delivery

[markstory]

We've had some issues with outbox delivery timing out and getting backlogged. Outboxes are well suited to small payloads on low-latency RPC calls which webhooks are not. By splitting storage off from outboxes we can handle updating delivery records in a way that avoids keeping transactions open while network requests are being sent.

The separate storage model also allows us to simplify the queries we use to find messages that need to be delivered as we don't have to spend cycles attempting to coalesce records that cannot be coalesced.

These changes add the tasks to process webhooks with this new storage model and include an option to control rollout.

[codecov[bot]]

Codecov Report

Attention: 16 lines in your changes are missing coverage. Please review.

Comparison is base (d549f7e) 81.39% compared to head (c9ca884) 81.40%. Report is 3 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff            @@
##           master   #64395    +/-   ##
========================================
  Coverage   81.39%   81.40%            
========================================
  Files        5252     5254     +2     
  Lines      232183   232326   +143     
  Branches    45598    45619    +21     
========================================
+ Hits       188990   189120   +130     
- Misses      37304    37314    +10     
- Partials     5889     5892     +3     

Files	Coverage Δ
src/sentry/conf/server.py	88.18% <ø> (ø)
src/sentry/hybridcloud/models/webhookpayload.py	100.00% <100.00%> (ø)
src/sentry/hybridcloud/tasks/__init__.py	100.00% <100.00%> (ø)
...rc/sentry/middleware/integrations/parsers/slack.py	90.69% <ø> (ø)
src/sentry/options/defaults.py	100.00% <100.00%> (ø)
src/sentry/testutils/factories.py	93.41% <100.00%> (ø)
src/sentry/middleware/integrations/parsers/base.py	87.12% <75.00%> (-0.17%)	:arrow_down:
src/sentry/hybridcloud/tasks/deliver_webhooks.py	94.00% <94.00%> (ø)
src/sentry/testutils/outbox.py	71.05% <75.00%> (+1.82%)	:arrow_up:

... and 7 files with indirect coverage changes

[sentry[bot]]

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

• ‼️ SiloClientError: Cannot invoke RegionSiloClient from MONOLITH. Only available in: CONTROL sentry.hybridcloud.tasks.deliver_webhooks.drain... View Issue

_{Did you find this useful? React with a 👍 or 👎}