nrtsearch icon indicating copy to clipboard operation
nrtsearch copied to clipboard

Restore broken if running primary/replica on same host

Open umeshdangat opened this issue 5 years ago • 0 comments

When we run both primary and replica on the same node, back them up and try to restore them on same node, the restored indexDir (index data) path ends ups pointing to same directory on disk.

Steps to repo

  1. Backup primary (and replica state)
  2. Down primary
  3. Down secondary
  4. Bring up new primary and restore it using resourceName in 1
  5. Bring up new secondary on same host as 4 and restore it using resourceName in 2

Root cause We back up index data from primary via the BackupIndex command. StateDir itself is also backed up and has to be restored before actual data. Upon restoring actual we reuse the indexDirectory specified in the stateDir. Then we download data to a new location. Finally, we simply create a symlink to point to the downloaded location of index data from the indexDir specified in stateDir.

The indexDir path looks like below: serverPrimary@ -> /var/folders/xl/8yyfg7s93k95g3vtr04b8z9jpd3vss/T/junit15797558079443604738/archiver/testresource_data/current/serverPrimary

Issue is when we run a replica/secondary on the same host we create another symlink to the same location of data: serverSecondary@ -> /var/folders/xl/8yyfg7s93k95g3vtr04b8z9jpd3vss/T/junit15797558079443604738/archiver/testresource_data/current/serverPrimary

Thus both primary and secondary/replica now use same underlying files. This is not an issue typically since we generally spin up different containers for restoring and running primary and replica.

umeshdangat avatar Feb 05 '20 00:02 umeshdangat