fix(deletions): Fix BulkDeleteQuery pagination
BulkDeleteQuery uses RangeQuerySetWrapper to iterate over results use a non-unique fields - last_seen. This actually doesnt work and skips values on rows with same last_seen. The only way to solve this and paginate correctly is to paginate over a complex combined condition of both last_seen and id.
To do this RangeQuerySetWrapper has been adapted to support a list to order_by, expecting this list to end with a unique key. Added tests and converted BulkDeleteQuery to use it.
Codecov Report
:x: Patch coverage is 98.73418% with 1 line in your changes missing coverage. Please review.
:white_check_mark: All tests successful. No failed tests found.
| Files with missing lines | Patch % | Lines |
|---|---|---|
| src/sentry/db/deletion.py | 90.90% | 1 Missing :warning: |
Additional details and impacted files
@@ Coverage Diff @@
## master #104435 +/- ##
============================================
+ Coverage 63.54% 80.55% +17.00%
============================================
Files 9347 9348 +1
Lines 399913 400452 +539
Branches 25646 25615 -31
============================================
+ Hits 254128 322587 +68459
+ Misses 142986 77417 -65569
+ Partials 2799 448 -2351
@wedamija alright refactored quite a bit - still left result_value_getter as not always returning a dict since that would force changing every call to RangeQuerySetWrapper across the entire codebase which felt like too big of a disruption, and also very specific since I need every place to know to write {"pk": item} which is cumbersome.