backup-utils
backup-utils copied to clipboard
github
Feature/custom restore data dir
This provides an option for the backup/restore utilities to use separate base directories
Based on a customer use case:
When each backup process is complete, the data is sent off to HDFS for storage and the local copies are removed (the whole backup folder, including snapshot subfolders). When a restore from a backup is requested, the backup data is retrieved from the HDFS and placed back into the 'GHE_DATA_DIR', this is required, due to the fact that the backup and restore scripts refer to the same 'GHE_DATA_DIR'.
This scenario causes conflicts if a restore is attempted while a backup is still in progress.
This feature provides an option for the backup/restore utilities to use separate base directories. It introduces GHE_RESTORE_DATA_DIR. Allowing users to restore from a different data location, then the backup.
All original Backup/Restore functionality (cmd options) remain the same, this simply adds additional flexibility. If no GHE_RESTORE_DATA_DIR is provided in the backup.config file, the original GHE_DATA_DIR will be used.
This was "life" tested with the GitHub-Demo-Stack (AWS) and an additional EC2 Restore Instance
Note: To address the customer's use case, alternative solutions, that don't require a code change, could have been used.
For example, run " a different backup-utils installation" (for restore actions), or don't back up and delete the whole backup folder, only individual snapshots, etc.
The decision to add more flexibility to our tools comes from the fact that this would not require the customer(s) to adjust their process (or policy) and that this adds additional flexibility without interfering with the existing functionality.
Test Runs
$ ./test/test-ghe-restore.sh
test: ghe-restore-snapshot-path reports an error when current symlink doesn't exist ... OK (0s)
test: ghe-restore-snapshot-path reports an error when specified snapshot doesn't exist ... OK (0s)
test: ghe-restore into configured vm ... OK (9s)
test: ghe-restore logs the benchmark ... OK (23s)
test: ghe-restore aborts without user verification ... OK (0s)
test: ghe-restore accepts user verification ... OK (5s)
test: ghe-restore -c into unconfigured vm ... OK (10s)
test: ghe-restore into unconfigured vm ... OK (28s)
test: ghe-restore with host arg and config value ... OK (14s)
test: ghe-restore with host arg ... OK (10s)
test: ghe-restore no host arg or configured restore host ... OK (0s)
test: ghe-restore with no pages backup ... OK (8s)
test: ghe-restore cluster backup to non-cluster appliance ... OK (0s)
test: ghe-restore no leaked ssh host keys detected ... OK (0s)
test: ghe-restore with current backup leaked key detection ... OK (0s)
test: ghe-restore fails when restore to an active HA pair ... OK (1s)
test: ghe-restore honours --version flag ... OK (0s)
test: ghe-restore honours --help and -h flags ... OK (0s)
test: ghe-restore exits early on unsupported version ... OK (0s)
test: ghe-restore cluster ... OK (5s)
test: ghe-restore missing directories or files from source snapshot displays warning ... OK (6s)
$ ./test/test-ghe-backup.sh
test: ghe-backup first snapshot ... OK (4s)
test: ghe-backup subsequent snapshot ... OK (5s)
test: ghe-backup logs the benchmark ... OK (5s)
test: ghe-backup with relative data dir path ... OK (4s)
test: ghe-backup fails fast when old style run in progress ... OK (1s)
test: ghe-backup cleans up stale in-progress file ... OK (4s)
test: ghe-backup without management console password ... OK (4s)
test: ghe-backup empty hookshot directory ... OK (3s)
test: ghe-backup empty git-hooks directory ... OK (4s)
test: ghe-backup fsck ... OK (15s)
test: ghe-backup stores version when not run from a clone ... OK (4s)
test: ghe-backup with leaked SSH host key detection for current backup ... OK (4s)
test: ghe-backup with no leaked keys ... OK (3s)
test: ghe-backup honours --version flag ... OK (0s)
test: ghe-backup honours --help and -h flags ... OK (0s)
test: ghe-backup exits early on unsupported version ... OK (1s)
test: ghe-backup-strategy returns rsync for HA backup ... OK (0s)
test: ghe-backup cluster ... OK (4s)
test: ghe-backup not missing directories or files on source appliance ... OK (5s)
test: ghe-backup missing directories or files on source appliance ... OK (4s)
test: ghe-backup fix_paths_for_ghe_version performance tests - gists ... OK (0s)
test: ghe-backup fix_paths_for_ghe_version performance tests - wikis ... OK (0s)
test: ghe-backup fix_paths_for_ghe_version newer/older ... OK (1s)
/cc @michaelsainz @steffen
@maclarel thanks for the feedback ...I'll address the merge conflict and we should be good to go for another 👀
@jefeish are you still planning on addressing the merge conflict? or is the current one a new merge conflict? 🙈
@jefeish are you still planning on addressing the merge conflict? or is the current one a new merge conflict? 🙈
@cainejette yes, I am still planning to address this ... I just got totally pulled in a different direction ...sorry 😞
yes, I am still planning to address this ... I just got totally pulled in a different direction ...sorry 😞
no problem! just checking in as I was doing a pass over notifications :)
Hi @jefeish! Apologies for the delayed response. It's been a while since this was open, is this still an issue that you are looking to get merged into the product?
