OpenSearch
OpenSearch copied to clipboard
[BUG] Handle delete cases with star tree index
Describe the bug
- Refresh fails when delete documents are performed
"reason": {
"type": "refresh_failed_engine_exception",
"reason": "Refresh failed",
"index": "logs-241998",
"shard": "0",
"index_uuid": "d02mxLclQrmb65qHY8i6fg",
"caused_by": {
"type": "file_already_exists_exception",
"reason": "/Users/gbh/Documents/dev/OpenSearch/build/testclusters/runTask-0/data/nodes/0/indices/d02mxLclQrmb65qHY8i6fg/0/index/_6.cidvd"
}
}
- Merge gives inaccurate results when there are deleted documents
Related component
Search:Performance
To Reproduce
Perform refresh
and merge
operations with deleted documents.
Expected behavior
- Refresh / Lucene flush should work with star tree index irrespective of deleted documents. In order to do this, fix is idenitfied to include
SegmentSuffix
as part of custom write state and read state used during star tree file formats - For merge , there are few cases where
SegmentDocValuesProducer
is present asDocValuesProducer
in a segment instead of theCodecDocValuesProducer
. So one change is to pick DocValuesProducer in the DVP list where star tree index is present - each field is associated with only one of the DocValuesProducer - even SegmentDocValuesProducer has a map of FieldInfo to DocValuesProducer which confirms the same. SoStarTreeValues
composite field too will be present in only one of theDocValuesProducers
Also, star tree index will not account for deleted documents except for cases where complete segments are dropped during segment merge. [ In which case star tree associated with that segment is also dropped ] During merge, out of all the segments which does have star tree , we will merge based on the values already present in star tree index - hence deletes will not be accounted.
This issue fix handles the bug around enforcing the above behavior.
Additional Details
Plugins Please list all plugins currently enabled.
Screenshots If applicable, add screenshots to help explain your problem.
Host/Environment (please complete the following information):
- OS: [e.g. iOS]
- Version [e.g. 22]
Additional context Add any other context about the problem here.