OpenSearch icon indicating copy to clipboard operation
OpenSearch copied to clipboard

[BUG] Handle delete cases with star tree index

Open bharath-techie opened this issue 4 months ago • 0 comments

Describe the bug

  1. Refresh fails when delete documents are performed
"reason": {
                    "type": "refresh_failed_engine_exception",
                    "reason": "Refresh failed",
                    "index": "logs-241998",
                    "shard": "0",
                    "index_uuid": "d02mxLclQrmb65qHY8i6fg",
                    "caused_by": {
                        "type": "file_already_exists_exception",
                        "reason": "/Users/gbh/Documents/dev/OpenSearch/build/testclusters/runTask-0/data/nodes/0/indices/d02mxLclQrmb65qHY8i6fg/0/index/_6.cidvd"
                    }
                }
  1. Merge gives inaccurate results when there are deleted documents

Related component

Search:Performance

To Reproduce

Perform refresh and merge operations with deleted documents.

Expected behavior

  1. Refresh / Lucene flush should work with star tree index irrespective of deleted documents. In order to do this, fix is idenitfied to include SegmentSuffix as part of custom write state and read state used during star tree file formats
  2. For merge , there are few cases where SegmentDocValuesProducer is present as DocValuesProducer in a segment instead of the CodecDocValuesProducer. So one change is to pick DocValuesProducer in the DVP list where star tree index is present - each field is associated with only one of the DocValuesProducer - even SegmentDocValuesProducer has a map of FieldInfo to DocValuesProducer which confirms the same. So StarTreeValues composite field too will be present in only one of the DocValuesProducers

Also, star tree index will not account for deleted documents except for cases where complete segments are dropped during segment merge. [ In which case star tree associated with that segment is also dropped ] During merge, out of all the segments which does have star tree , we will merge based on the values already present in star tree index - hence deletes will not be accounted.

This issue fix handles the bug around enforcing the above behavior.

Additional Details

Plugins Please list all plugins currently enabled.

Screenshots If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

  • OS: [e.g. iOS]
  • Version [e.g. 22]

Additional context Add any other context about the problem here.

bharath-techie avatar Oct 18 '24 10:10 bharath-techie