syncoid exclude snapshot types by name

[graham00]

I've made some changes to add a new command, --exclude-snaps, so that I can avoid replicating certain types of snapshots. For example:

syncoid --exclude-snaps="hourly"

This skips snapshots in getsnaps() matching that term, and it then loops syncdataset with -i from oldest matching snapshot to next-oldest non-excluded snapshot until it gets a return value indicating it reached the last snapshot. Seems to be cutting the amount of replicated data for most of my VMs over WAN down to less than 50% of what it was.

I can make a pull request with the changes, but if you're not wanting to approach this issue from that direction, then I'll probably just add it in my rsync fork as a personal workaround for now. Thanks

Edit - I did end up adding support for excluding to my fork.

[sithson]

Hi Graham and jimsalterjrs.

I would also love to see the --exclude functionality as I don't need to replicate hourly snapshots to my backup site.

As of now, hourly snaps always replicate, then sanoid removes them every minute on the backend site. Very annoying, and causes unnecessary stress on my backup disks.

Thanks for a great tool jimsalterjrs, and I hope you consider Grahams request =) Cheers!

[phreaker0]

@graham00 did you create a branch/pr anywhere with your changes?

[redmop]

Does Sanoid handle not trying to delete snapshots that are cloned? That would be a handy addition.

[phreaker0]

@redmop zfs destroy will error out in this case by default:

zfs destroy test/a@test
cannot destroy 'test/a@test': snapshot has dependent clones
use '-R' to destroy the following datasets:
test/clone

[redmop]

So I'll receive this error every time sanoid runs until excluded or it can actually delete it. Maybe a patch to include an option to ignore this error?

[phreaker0]

@redmop A patch to silence would be possible but this would need to be configurable which should default to off. In the normal use case if you instruct sanoid to prune specific snapshots you want to know if it failed.

[redmop]

Default to off, of course. I could always just rename the snapshot when I clone it too.

[phreaker0]

@graham00 ping

[graham00]

@graham00 did you create a branch/pr anywhere with your changes?

@phreaker0 I did - see the link to my fork in my opening comment. The usage for excluding hourly snapshots is documented. If you're only after that, then I suggest you also add "--no-rsync" as is described above that section, so that my fork otherwise behaves like the original syncoid does. (btw - sorry I didn't catch your comment in September!)

Also - I was occasionally syncing my fork up with changes to the main syncoid, but I haven't for a while. My fork is currently based on syncoid-1.4.16, so any feature in normal syncoid beyond that won't work in my fork.

[phreaker0]

@graham00 thx, just FYI syncoid now supports resume able replication.

[matveevandrey]

Would be happy to see this functionality too

[dbuades]

+1 for this feature too. Have you considered making a pull request with the changes in your fork, @graham00 ?

[jackmurray]

+1, would love to see this

[amartin3225]

+1

[Ratief]

This is a very important feature for me as it would save a lot of wasted/useless data transfer.

[ryanjaeb]

I could also make use of this. I re-use the snapshot name @borgsnap to create a stable location for running Borg backups. When those snapshots end up on my syncoid targets, it causes future syncs to fail. For now it's easy enough to prune those snapshots before running syncoid, but it would great if I could just exclude them in the first place via --exclude-snaps=borgsnap.

[redmop]

@ryanjaeb Might I suggest using the daily/weekly/monthly snapshot for borg backups? That's how I'm doing it.

[phreaker0]

@ryanjaeb I just list the latest snapshot and use it for borg backup.

[ryanjaeb]

@redmp @phreaker0 How do you ensure those snapshots aren't reaped during a long running Borg backup?

[redmop]

That is why I use dailies. I keep them for a 30 days. My frequents don't always last long enough to be backed up during the backup window.

[phreaker0]

@ryanjaeb I don't need to, zfs destroy won't work if the snapshot is mounted:

$ sudo mount -t zfs test@test /mnt/tmp/
$ sudo zfs destroy test@test
cannot destroy snapshot test@test: dataset is busy

[pquan]

There's an example of how to exlude specific snapshots in a fork of syncoid here: https://github.com/graham00/syncoid-rsync Unfortunately that fork is not maintained any more, but still usefull I hope.

Exclude hourly snapshots on remote backups saves a huge amount of data

[phreaker0]

@pquan it's drifted to much apart, syncoid handles much more (error) cases now. I'm planning to implement this but I don't know when :-)

[discostur]

+1 would be really cool to exclude specific snapshots from syncing

[graham00]

Sorry for the delayed response! Life has been crazy lately.

@dbuades

Have you considered making a pull request with the changes in your fork, @graham00 ?

I haven't, since I discussed it with Jim beforehand and he wanted to wait out official resume support. Understandable, since what I did was a fairly ugly (uses a lot of shell tools on both ends) implementation...though of course it does work lol.

The other more minor changes (exceptions/etc) could be merged more easily - honestly, I'm just severely lacking time in life so I haven't had the time to. Anyone should feel free to do so though! The code for the non-resume stuff should be pretty easy to merge back in.

@pquan

fork is not maintained any more

I just updated it with a new security-related feature (--no-rollback) that I've been testing personally for a while offline, but yeah, updates are far and few between and the newer syncoid checks for more error conditions, as phreaker0 mentioned. Basically, I just hack quick changes into it now and then as I personally need something. Anyone should feel free to diff it out and add the code to the main branch if helpful though! Honestly, part of it is that I'm not particularly familiar with how to do pull/etc stuff on github - this branch and whatnot was the first time I've ever done anything beyond download from github.

[yanky83]

+1

[thorro]

+1

EDIT: As I needed the function ASAP, I just added the following (hackish) code to syncoid getsnaps() function, in each of the two foreach blocks.

            if ($line =~ /$fs\@__replicate.*/) {
                next;
            }

This will skip all snapshots starting with __replicate.

[phreaker0]

@thorro but this will only work if __replicate is always the most recent snapshot or you are using --no-stream option.

[mreymann]

Any news on the --exclude-snaps feature? Would be nice to have!

[RubyFeinstein]

Just opened a pull request with the required change, let me know if I missed anything: https://github.com/jimsalterjrs/sanoid/pull/593