Expunge big segment with oversize deletePct caused by continuously updating a batch of data

[MichelLiu]

Continuously updating a batch of data can cause the great grow up deletePct of the big segments which can slow down the search performance.

[jpountz]

This PR is mixing a force-merge-specific setting with natural merges.

Can you give more context about the problem that you are trying to solve? Is setting deletesPctAllowed to a low-ish value like 20 not good enough?

[MichelLiu]

I had a problem with the tiered merge policy. As I continuously updated a batch of data over time and time, then I got a lot of segments with 4.9G which segDelPct already greater than deletePctAllowed and cannot be merged by tiered merge policy. Then I found the code here and figured out the reason: if (segSizeDocs.sizeInBytes > maxMergedSegmentBytes / 2 && (totalDelPct <= deletesPctAllowed || segDelPct <= deletesPctAllowed)) { iter.remove(); tooBigCount++; // Just for reporting purposes. totIndexBytes -= segSizeDocs.sizeInBytes; allowedDelCount -= segSizeDocs.delCount; }

Here was the segments I had met before:

1613741580098 0 p 10.10.112.123 _2h 89 1224440 569330 4.9gb 4905832 true true 8.4.0 false 1613741580098 0 p 10.10.112.123 _4v 175 2383463 425919 4.9gb 5636245 true true 8.4.0 false 1613741580098 0 p 10.10.112.123 _6n 239 2891298 380212 4.9gb 5617940 true true 8.4.0 false 1613741580098 0 p 10.10.112.123 _1lwc 75036 468350 364104 4.3gb 3718611 true true 8.4.0 false 1613741580098 0 p 10.10.112.123 _1xh2 90038 678187 252779 3.6gb 3453739 true true 8.4.0 false 1613741580098 0 p 10.10.112.123 _25u8 100880 482795 237275 4.1gb 3370799 true true 8.4.0 false 1613741580098 0 p 10.10.112.123 _2fld 113521 721503 225160 4.1gb 3776954 true true 8.4.0 false 1613741580098 0 p 10.10.112.123 _2m9h 122165 831574 127572 4.2gb 3812013 true true 8.4.0 false 1613741580098 0 p 10.10.112.123 _2n01 123121 34000 27437 345.3mb 543426 true true 8.4.0 true 1613741580098 0 p 10.10.112.123 _2nq6 124062 36985 19838 319.2mb 515882 true true 8.4.0 true 1613741580098 0 p 10.10.112.123 _2o7d 124681 52725 40581 556.3mb 632128 true true 8.4.0 true 1613741580098 0 p 10.10.112.123 _2ouj 125515 11158 6330 114mb 235396 true true 8.4.0 true

And I had an index with 564G, after bulk updating for a month, then grows up to 1400G. That caused significant waste of disk, and also highed up the search delay to 450ms. So we have to reindex the index per month now.

My solution is to merge the large segments as low-frequency as possible.

[jpountz]

If I read this line correctly, it says that large segments (more than 50% the maximum segment size) shouldn't be merged unless both the percentage of deletes of the segment and the percentage of deletes across the entire index are greater than the maximum percentage of deletes. Maybe there is an issue with large segments and large numbers of deletions, but this is not the one?

[jpountz]

If you look at BaseMergePolicyTestCase and TestTieredMergePolicy, we actually have tests that simulate merges in order to verify that things like the maximum percentage of deletes work correctly (see BaseMergePolicyTestCase#doTestSimulateUpdates in particular). If you could write a test that demonstrates the problem that you're observing on your cluster with large segments, that would be super helpful.

[MichelLiu]

This PR is mixing a force-merge-specific setting with natural merges.

Can you give more context about the problem that you are trying to solve? Is setting deletesPctAllowed to a low-ish value like 20 not good enough?

Setting deletesPctAllowed to a low-ish value will slow down search response, waste disk io

[MichelLiu]

If you look at BaseMergePolicyTestCase and TestTieredMergePolicy, we actually have tests that simulate merges in order to verify that things like the maximum percentage of deletes work correctly (see BaseMergePolicyTestCase#doTestSimulateUpdates in particular). If you could write a test that demonstrates the problem that you're observing on your cluster with large segments, that would be super helpful.

This is not easy to simulate this case, but I had a lot monitoring data.Fig. out I will try to consider the way to simulate later.

[github-actions[bot]]

This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the [email protected] list. Thank you for your contribution!