pinot icon indicating copy to clipboard operation
pinot copied to clipboard

Improve realtime Lucene text index freshness/cpu/disk io usage

Open itschrispeck opened this issue 8 months ago • 3 comments

This PR allows for better freshness/cpu/disk io usage for realtime Lucene text index.

User facing changes:

  1. Add config pinot.server.lucene.min.refresh.interval.ms (default is 10, as 10ms was the previous behavior)
  2. Add config pinot.server.lucene.max.refresh.threads (default is 1, as a single thread was the previous behavior)

Implementation changes:

  1. Use scale-first ScalingThreadPoolExecutor to allow for multiple background refresh threads to refresh Lucene indexes
    1. All RealtimeLuceneTextIndex._searcherManagers are evenly distributed between background refresh threads.
    2. The refresh thread pool is 1 thread:1 RealtimeLuceneTextIndex, up to max threads configured, then each thread handles multiple RealtimeLuceneTextIndex
    3. If tables are deleted/consuming segment rebalance occurs leaving a thread without a RealtimeLuceneTextIndex to refresh, the thread will be removed
  2. Refactor RealtimeLuceneTextIndex specific logic out of MutableSegmentImpl - the index itself registers itself with the refresh manager, and is removed once closed
  3. Add LuceneNRTCachingMergePolicy to perform best effort merging of in-memory Lucene segments - each refresh causes a flush, and making refreshes more common will cause huge numbers of small files.

With configs not set/default settings, we see lower cpu/disk io/slightly better index freshness. With more aggressive configs, we see much better index freshness (we have many tables w/ text index) at the same or similar resource usage.

For testing, we've had this deployed in some of our prod clusters for a bit without issues.

itschrispeck avatar Jun 27 '24 21:06 itschrispeck