John Leach

Major phases of Lucene Query for data access: 1. IndexSearcher Open 2. Term Weights*** 3. IndexSearcher Probe 4. Scoring Weighting Here is the call stack in the code… `com.apple.foundationdb.record.lucene.directory.FDBIndexInput.readBlock(FDBIndexInput.java:130) at...

Major phases of Lucene Query for data access: 1. IndexSearcher Open 2. Term Weights 3. IndexSearcher Probe 4. Scoring Weighting*** ** 1 Sequential call per Leaf `com.apple.foundationdb.record.lucene.directory.FDBDirectory.readBlock(FDBDirectory.java:357) at com.apple.foundationdb.record.lucene.directory.FDBIndexInput.readBlock(FDBIndexInput.java:130) at...

Parallelize openInputs and remove footer checks in BlockTreeTermsReader #1688

9

This removes the footer checks and parallelizes two data fetches for each opening of a segment reader.

The BlockTreeTermsReader currently blocks (no pun attended) 4 times in a row and fetches additional blocks for each segment file. The goal is to remove 2 of the calls and...

The current system sequentially opens the readers. `com.apple.foundationdb.record.lucene.directory.FDBIndexInput.length(FDBIndexInput.java:197) org.apache.lucene.codecs.CodecUtil.validateFooter(CodecUtil.java:513) org.apache.lucene.codecs.CodecUtil.retrieveChecksum(CodecUtil.java:490) org.apache.lucene.codecs.CodecUtil.retrieveChecksum(CodecUtil.java:509) org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.(BlockTreeTermsReader.java:240) com.apple.foundationdb.record.lucene.codec.LuceneOptimizedPostingsFormat.fieldsProducer(LuceneOptimizedPostingsFormat.java:57) org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:114) org.apache.lucene.index.SegmentReader.(SegmentReader.java:83) org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:69) org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:61) org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:720) org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:84) org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:64) com.apple.foundationdb.record.lucene.directory.FDBDirectoryWrapper.getReader(FDBDirectoryWrapper.java:85) com.apple.foundationdb.record.lucene.directory.FDBDirectoryManager.getIndexReader(FDBDirectoryManager.java:119) com.apple.foundationdb.record.lucene.LuceneRecordCursor.getIndexReader(LuceneRecordCursor.java:292) com.apple.foundationdb.record.lucene.LuceneRecordCursor.performScan(LuceneRecordCursor.java:297)`

Transition SortedDocValuesField to BinaryPoint to support efficient deletes in Lucene #1595

7

The SortedDocValuesField is known to have issues deleting large number of records and requires the index to read more data than needed during a delete. The BinaryPoint implementation is known...

The large copy bytes blocks for each block sequentially.