OpenSearch
OpenSearch copied to clipboard
Fix assertion failure while deleting remote backed index
Description
- When an instance of Store is created, a
shardlock
is created which is released on closing the instance of store. - Currently, we create 2 instances of store for remote store backed indices:
store
andremoteStore
. - As there can be only one
shardlock
acquired for a given shard, the lock is shared betweenstore
andremoteStore
. - This creates an issue when we are deleting the index as it results in closing both
store
andremoteStore
. - In this PR, we create a different lock for remoteStore. As part of https://github.com/opensearch-project/OpenSearch/issues/13075, we want to move away from keeping 2 store instances.
- To reproduce the issue (without the change in this PR), we need to keep following sequence:
- Refresh Started and increased
store.ref
- Store.close - will not release shardLock as refCount is not 0
- RemoteStore.close - will release shardLock as the refCount is 0
- Refresh ended - will decrease
store.ref
, closes it and close fail as it finds that shard is not locked
- Refresh Started and increased
Related Issues
- https://github.com/opensearch-project/OpenSearch/issues/13871
- https://github.com/opensearch-project/OpenSearch/issues/13737
Check List
- [x] Functionality includes testing.
- [ ] ~API changes companion pull request created, if applicable.~
- [ ] ~Public documentation issue/PR created, if applicable.~
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.