quickwit icon indicating copy to clipboard operation
quickwit copied to clipboard

Merge policy with constant write amplification

Open fulmicoton opened this issue 3 years ago • 0 comments

The current merge policy attempts to merge splits spanning over contiguous window of time, with the same size, in groups of 10, to reach a given target size of 10 millions documents.

If we enable tag partitioning, we will end up creating small splits.

As a simple solution, we can add a configurable option to limit the number of merge a split can undergo. On a stable indexing throughput, this would be a nice approximation of a merge policy that targets the creation of a single split for a given amount of time.

fulmicoton avatar Jul 25 '22 08:07 fulmicoton