feat(tasks) Add metrics to track worker queue sizes
I'd like to better understand how 'full' the multiprocessing queues are within workers. By measuring additions into the child tasks and removals from the results, I hope to better understand how 'full' workers are, as the current empty queue metrics only show us when the queues are empty.
The new counter is modified in multiple threads, but because the operations are simple inc/dec I don't think the impacts of races will be that important.
Codecov Report
:x: Patch coverage is 85.71429% with 1 line in your changes missing coverage. Please review.
:white_check_mark: All tests successful. No failed tests found.
| Files with missing lines | Patch % | Lines |
|---|---|---|
| src/sentry/taskworker/worker.py | 85.71% | 1 Missing :warning: |
Additional details and impacted files
@@ Coverage Diff @@
## master #104394 +/- ##
===========================================
- Coverage 80.57% 80.54% -0.04%
===========================================
Files 9345 9345
Lines 399467 399991 +524
Branches 25597 25597
===========================================
+ Hits 321884 322177 +293
- Misses 77123 77354 +231
Partials 460 460
The new counter is modified in multiple threads, but because the operations are simple inc/dec I don't think the impacts of races will be that important.
I don't think this will be true for loaded workers (e.g. ingest-transactions etc.). Worse, I don't think we will actually know whether the metrics are accurate or not. I am reluctant to introduce metrics that we can't "trust".