quickwit icon indicating copy to clipboard operation
quickwit copied to clipboard

make delete task hammer less the metastore

Open trinity-1686a opened this issue 1 year ago • 2 comments

currently on startup, the delete task does quite a few request to the metastore. We could optimize a bit so less calls are made, and also make it so they are not made all at once. Here are some ideas:

  • DeleteTaskPipeline::spawn_pipeline could be called while holding a semaphore so there aren't 1k requests all at once.
  • get_relevant_stale_splits() could be made to accept None as delete opstamp, and figure itself what is the latest opstamp and return it along, saving one call to last_delete_opstamp() per index.
  • in partition_splits_by_deletes, call list_delete_tasks() only once, and sort ourselves which delete have already been applied to any given split

trinity-1686a avatar Feb 09 '24 16:02 trinity-1686a

Can we just reorder operations to make sure nothing happens if we don't have any delete operations to begin with?

fulmicoton avatar Feb 14 '24 02:02 fulmicoton

I think at the bare minimum, we need to call last_delete_opstamp() for each index (to check it is zero), or in a future where list_stale_splits() supports None, one call to that per index. Or we need a new metastore call which can handle multiple (all?) indexes at once

trinity-1686a avatar Feb 14 '24 10:02 trinity-1686a