jimsalterjrs
Add --include-snaps and --exclude-snaps options to syncoid
This PR adds new options to syncoid that filter out snapshots from the source and prevent their transfer to the destination. I added an --exclude-snaps option and an --include-snaps option; both accept a regex argument and both can be specified multiple times. If both options are used in the same syncoid call, then snapshots matching both patterns will be excluded.
In contrast to PR #632, using --exclude-snaps or --include-snaps does not imply --no-stream. That lets you start with a set of source snapshots like this:
- daily1
- hourly1
- hourly2
- daily2
- hourly3
- hourly4
- daily3
And end up with these snaps on the destination by using --exclude-snaps='hourly':
- daily1
- daily2
- daily3
In other words, using one of the new options causes syncoid to simulate a -I sync by doing a series of -i syncs between intermediate snapshots that are not filtered.
A note on the print statement refactoring: I wanted to remove the many if (debug) and if (!quiet) statements, but it does make this diff a lot bigger. Maintainers, please let me know if you'd like me to rebase this without commit 79aaf45.
I realized that my first push had some changes in the wrong commit, so I rebased to fix that.
One thing you should clarify in the docs is whether the regex is matched against only the snapshot name or the entire string of dataset@snapshot. It might also be good to add similar documentation to the --exclude option since there's now more than one.
@DarwinAwardWinner good point about the potential confusion between --exclude and the new options. I pushed another commit that adds a new --exclude-datasets option to replace --exclude, and prints out a deprecation warning when --exclude is used. I also added more to the README that hopefully clears up what the regex patterns are matched against.
Note that you have a typo in a few places: rexeg instead of regex.
Other than that, it looks good!
Oops, thanks for catching that typo @DarwinAwardWinner. I edited the last commit to fix that.
I've been testing this PR heavily on multiple ZFS hosts for the last 3 or 4 weeks. The new exclude functionality adds the only feature I was missing from sanoid/syncoid.
Good work @mr-vinn
Running for one week now - no errors so far :)
Regards Kilian
Am 08.01.2022 um 04:48 schrieb Ryan C. Thompson @.***>:
I've also been using this branch on my systems as well.
— Reply to this email directly, view it on GitHubhttps://github.com/jimsalterjrs/sanoid/pull/699#issuecomment-1007879175, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAT6Q7EJBZNJMW4OFHOVCJLUU6XYPANCNFSM5JDCCNNA. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you commented.Message ID: @.***>
This would be a fantastic feature to have. Any chance this is getting merged soon?
Any updates on when this could get merged? I am also very excited about this functionality
I just started using this branch for a couple datasets where I'm storing a lot of frequently snapshots on the host, but don't need them on the backup target. It's really great!
I'd like to suggest including some indication of the $sourcefs during the send/recv. One dataset I backup has a lot of child datasets and it's never clear to me which dataset is currently being operated on. Possible locations include having $sourcefs in either the a top-level output per dataset or with each incremental sync. Thanks.
@mr-vinn i rebased your PR to the latest master to resolve conflicts. This looks really great and is quite a huge change but the restructuring looks really clean. Unfortunately some tests fail.
I guess I'll take over here because the original author is inactive and this is a highly requested feature. I rebased the PR to the latest master to resolve conflicts. I also found the refactoring error which lead to replication failures in some cases, all tests run through now so this looks ready for prime time now.
I hope it gets some broader testing in the master. I will roll it out over my systems in the next time to make sure there is no breakage.
I hope it gets some broader testing in the master. I will roll it out over my systems in the next time to make sure there is no breakage.
At the very least, I end up testing the master branch myself--whenever I deploy a new system, I use master, not the latest release. Mostly for this exact reason. :)
Finally! Thanks for merging this at last. I'm already building+deploying master to some dev systems where they will be heavily tested - a mix of Ubuntu and Arch systems, x86_64 and aarch64. I shall report back.
This is long since closed but nonetheless after two weeks I can report back that testing has gone flawlessly. Thanks again for pushing this over the finish line at last.