OpenSearch icon indicating copy to clipboard operation
OpenSearch copied to clipboard

Fix assertion failure while deleting remote backed index

Open sachinpkale opened this issue 7 months ago • 1 comments

Description

  • When an instance of Store is created, a shardlock is created which is released on closing the instance of store.
  • Currently, we create 2 instances of store for remote store backed indices: store and remoteStore.
  • As there can be only one shardlock acquired for a given shard, the lock is shared between store and remoteStore.
  • This creates an issue when we are deleting the index as it results in closing both store and remoteStore.
  • In this PR, we create a different lock for remoteStore. As part of https://github.com/opensearch-project/OpenSearch/issues/13075, we want to move away from keeping 2 store instances.
  • To reproduce the issue (without the change in this PR), we need to keep following sequence:
    • Refresh Started and increased store.ref
    • Store.close - will not release shardLock as refCount is not 0
    • RemoteStore.close - will release shardLock as the refCount is 0
    • Refresh ended - will decrease store.ref, closes it and close fail as it finds that shard is not locked

Related Issues

  • https://github.com/opensearch-project/OpenSearch/issues/13871
  • https://github.com/opensearch-project/OpenSearch/issues/13737

Check List

  • [x] Functionality includes testing.
  • [ ] ~API changes companion pull request created, if applicable.~
  • [ ] ~Public documentation issue/PR created, if applicable.~

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

sachinpkale avatar Jun 30 '24 13:06 sachinpkale