OpenSearch icon indicating copy to clipboard operation
OpenSearch copied to clipboard

[Segment Replication] Remote store integration high level design.

Open mch2 opened this issue 2 years ago • 0 comments

Before starting a POC for this integration we should start with a high level design.

Some questions that come to mind.

  1. How do we compute and send the list of segments to fetch from the store?
  2. How do we fetch the SegmentInfos byte[] required?
  3. Are there any changes to failover steps? Currently we commit on replicas during failover and replay from xlog, can we ensure the newly selected replica always has the latest segments from store before promotion?
  4. How would we guarantee segments are present before fetching from remote store?
  5. Should we use existing push segrep design or have replicas poll remote store for updates? If so, how would this work?

lower lvl questions:

  1. Do we still need commit points on replicas?
  2. Can we now disable fsyncs on primaries?

mch2 avatar Sep 19 '22 16:09 mch2