pioreactor icon indicating copy to clipboard operation
pioreactor copied to clipboard

Continuously backup the database instead of batch backups

Open CamDavidsonPilon opened this issue 3 years ago • 1 comments

Consider something like litestream

Pros:

  • no need for heavy backup jobs (#81)
  • always up to date data replicas, instead of every N days
  • delete our own internal backup code

Cons:

  • how does litestream deal with a replica being unavailable?
  • more cpu work upfront during an experiment
  • more overhead during an experiment

CamDavidsonPilon avatar Apr 25 '22 13:04 CamDavidsonPilon

One problem is that I don't know the hostname of a worker in the cluster apriori to setting up litestream. Also I can't gauruntee it's always available and online.

This could be solved by specifying it in the config.ini - maybe this is a plugin and not a builtin feature.


Alternatively, litestream could backup cont. locally to ~/.pioreactor/storage, and every N days, bulk back up to other machines.

CamDavidsonPilon avatar Apr 25 '22 15:04 CamDavidsonPilon