OpenSearch icon indicating copy to clipboard operation
OpenSearch copied to clipboard

[Meta] NoOp replication for remote translogs with NRT segment replication

Open ashking94 opened this issue 2 years ago • 0 comments

We have discussed NoOp replication on high level in #3706 and the proposal can be found here and the approach selected has been discussed here.

As part of the exercise, in brief, we want to achieve the following -

  • Use replication call for primary term validation, but the call itself is no op.
  • Since the replicas don't need the translog, recovery needs to be refactored for replica. We do not need to replay Lucene operations anymore. Similarly, there would be refactoring in replica-primary promotion and replica-replica recovery.
  • PRRL helps to keep a certain history of operations so that if a recovery was to happen, then the recovery can make use of replaying Lucene operations,
  • Global and local checkpoints are used in context of indexing & translog. Since the translog will be stored remotely, we are guaranteed to have translogs always in case of node failure and hence we have to decouple checkpoints interaction between primary and replica. Global / local checkpoint should hold no more relevance with remote translog.
  • Redefine insync shards now can be any replica since we have segments and translogs available remotely.

This issue is a meta-issue for tracking the progress of implementing no-op replication. We will proceed with implementing no-op replication along the following rough plan:

  • [x] #4318
  • [ ] #4502
  • [ ] Stop creation of empty translog in the cleanFiles method of RecoveryTarget. TODO - create issue.
  • [ ] #4503
  • [ ] #4504
  • [ ] #4505
  • [ ] #4506
  • [ ] Refactoring the sync/async calls from primary shards to replicas which modifies the checkpoints in the replica's replication tracker. TODO - create issue.
  • [ ] Disassociate store and translog for remote translog enabled cases. Refer to Store.associateIndexWithNewTranslog(translogUUID). TODO - create issue

ashking94 avatar Sep 14 '22 10:09 ashking94