quickwit
quickwit copied to clipboard
Merge policy with constant write amplification
The current merge policy attempts to merge splits spanning over contiguous window of time, with the same size, in groups of 10, to reach a given target size of 10 millions documents.
If we enable tag partitioning, we will end up creating small splits.
As a simple solution, we can add a configurable option to limit the number of merge a split can undergo. On a stable indexing throughput, this would be a nice approximation of a merge policy that targets the creation of a single split for a given amount of time.