loki
loki copied to clipboard
Merge delete requests table from multiple compactors
Describe the bug Compactor stores delete requests in a table with a single boltdb file and keep uploading that file to the storage every 5 mins. It would only download the delete requests file from the storage when it does not find it on the disk during startup. The delete requests feature was built assuming there would always be a single compactor running, so there was never a need to check for updates in the storage.
We added support for running multiple compactors in SSD mode, with just one of them performing the compaction using a ring. The problem here is that each instance would keep uploading its local file to the storage. So if one compactor got a delete request and uploaded it to the storage, another compactor would replace the file in storage with its local file(possibly empty if it never got any requests). To avoid overwriting each other's delete requests file, we should implement similar logic as chunk index, where we upload files with unique names and compact them down to a single file.
To Reproduce Steps to reproduce the behavior:
- Run 2 compactors.
- Send a delete request to just one compactor.
After >5 mins the file in the storage would have no delete requests.