sentry icon indicating copy to clipboard operation
sentry copied to clipboard

feat(tasks) Add metrics to track worker queue sizes

Open markstory opened this issue 1 month ago • 2 comments

I'd like to better understand how 'full' the multiprocessing queues are within workers. By measuring additions into the child tasks and removals from the results, I hope to better understand how 'full' workers are, as the current empty queue metrics only show us when the queues are empty.

The new counter is modified in multiple threads, but because the operations are simple inc/dec I don't think the impacts of races will be that important.

markstory avatar Dec 04 '25 16:12 markstory

Codecov Report

:x: Patch coverage is 85.71429% with 1 line in your changes missing coverage. Please review. :white_check_mark: All tests successful. No failed tests found.

Files with missing lines Patch % Lines
src/sentry/taskworker/worker.py 85.71% 1 Missing :warning:
Additional details and impacted files
@@             Coverage Diff             @@
##           master   #104394      +/-   ##
===========================================
- Coverage   80.57%    80.54%   -0.04%     
===========================================
  Files        9345      9345              
  Lines      399467    399991     +524     
  Branches    25597     25597              
===========================================
+ Hits       321884    322177     +293     
- Misses      77123     77354     +231     
  Partials      460       460              

codecov[bot] avatar Dec 04 '25 16:12 codecov[bot]

The new counter is modified in multiple threads, but because the operations are simple inc/dec I don't think the impacts of races will be that important.

I don't think this will be true for loaded workers (e.g. ingest-transactions etc.). Worse, I don't think we will actually know whether the metrics are accurate or not. I am reluctant to introduce metrics that we can't "trust".

evanh avatar Dec 08 '25 16:12 evanh