quickwit
quickwit copied to clipboard
make delete task hammer less the metastore
currently on startup, the delete task does quite a few request to the metastore. We could optimize a bit so less calls are made, and also make it so they are not made all at once. Here are some ideas:
DeleteTaskPipeline::spawn_pipelinecould be called while holding a semaphore so there aren't 1k requests all at once.get_relevant_stale_splits()could be made to accept None as delete opstamp, and figure itself what is the latest opstamp and return it along, saving one call tolast_delete_opstamp()per index.- in
partition_splits_by_deletes, calllist_delete_tasks()only once, and sort ourselves which delete have already been applied to any given split
Can we just reorder operations to make sure nothing happens if we don't have any delete operations to begin with?
I think at the bare minimum, we need to call last_delete_opstamp() for each index (to check it is zero), or in a future where list_stale_splits() supports None, one call to that per index. Or we need a new metastore call which can handle multiple (all?) indexes at once