Improved segment build time for Lucene text index realtime to offline conversion

[itschrispeck]

Problem: We typically see long (7-10min) segment build times when using Lucene index with 1-1.5GB segment sizes. 70-80% of this time is spent building the Lucene text index.

Background: In the existing implementation the Lucene index stores Pinot docIds: for the mutable segment these are the mutable segment's docIds, for the immutable segment we store each row with its new immutable segment docId. Lucene queries return the matching Lucene DocIds, and we compute these on the fly for the mutable index, or from a mapping file for the immutable index.

Change Summary: This change copies the mutable Lucene index during realtime segment conversion to reuse, instead of building a new Lucene index. To prepare for copying, a commit() method is added to the MutableSegment interface. To handle the potential docId change, sortedDocIds is added to IndexCreationContext to compute a temporary mapping between the mutable docId and the immutable segment's docId. This temporary mapping is used during segment conversion to build the mapping file between the Lucene docId and the new immutable segment's docId. This mapping file is built during segment conversion, instead of during segment load in the traditional path.

Internally we've seen roughly 40-60% improvement in overall segment build time. The lower peaks are from a table/tenant with this change, the higher ingestion delay peaks are from an identical table in a tenant without this change:

Testing: deployed internally, local testing, validated basic pause/restart/reload operations on a table to ensure no regression in TextIndexHandler index build.

tags: ingestion performance