Idea to prevent lack of common snapshots

[doctorjames]

This idea addresses the main concern I had in issue #48 (though that is still worth implementing even with this idea) and was prompted by reading issue #57.

What do you think about making pyznap send maintain a sequence of snapshots on each source file system that are not cleaned by pyznap snap --clean but only by pyznap send itself? The minimal version could work something like this:

On calling pyznap send:

1. create a snapshot: zfs snapshot tank/data@pyznap_2020-01-06_01:00:20_send
2. do actual zfs send
3. on success, list *_send snapshots on destination and verify existence of current *_send snapshot in list from destination (this is just belt and braces given the next step will require listing the snapshots anyway)
4. on successful verification, delete all but the current *_send snapshot on both destination and source

This transactional approach will prevent pyznap snap --clean inadvertently removing all common snapshots in the event of networking or remote disk space issues.

You might wish to encode the destination name in the snapshot name for setups sending to multiple locations.

Hopefully this improvement is also straightforward enough to implement using the existing snapshot making and cleaning functionality in pyznap?

[rbrewer123]

As the author of #57, I think the above idea is interesting and would provide additional assurance in some cases (I mention some below), but wouldn't have helped my situation.

My backup scheme involves 2 external backup pools which I periodically swap offsite. Although the pools are named "backup1" and "backup2", my import script imports them with a temporary name "backup". Then everything after that (including pyznap) doesn't know or care which one it is backing up to, scrubbing, or whatever. The above scheme would have deleted the "_send" snapshot on my sending pool and I would have had been back to "no common snapshots" from #57.

I think the tradeoff in the above scheme is providing more assurance of a common snapshot at the expense of a clearly defined retention policy. I think it would be most useful in cases where a short retention policy is desired, perhaps because the backup pool is not large enough to maintain a longer retention policy.

[yboetz]

Syncoid has that feature, it creates a send snapshot and rotates them after sending, so there is always at least one common snapshot. I had some problems with this though, e.g. when sending to multiple destinations or stale snapshots that didn't get cleaned up. That is why I built pyznap without this feature.

I always advice for long retention policies compared to the backup frequency, though a send snapshot might give some more assurance in some cases. The idea to encode destination name into the snapshot name is great, that would fix one of the problems I've had previously with syncoid. Implementing it should be possible with the current cleaning functionality, though I cannot make any promises if/when I'll work on this :).

[recklessnl]

+1 to this idea. @yboetz any updates on this feature?

[yboetz]

@recklessnl No updates on this feature, sorry. I haven't had much time to work on pyznap in the past few months, so feature requests are quite slow...