doudou
Add remote backup through SSH
Not really a bug, but more a place for discussion about what would be required for this. I just started using snapsync, but remote backups are what I want sooner or later.
What would be required to add this? How complicated would this be to add? I have never written a line of Ruby, but I might be able to help out anyway.
I would also love to have that feature, but don't have the time.
I would see two different possibilities here: a. performing backup on a machine that we control (i.e. where snapsync can be installed) b. performing backup on a machine that does not have snapsync, but does give us btrfs access
I'm not sure that (b) makes any sense, since one needs root access to create btrfs subvolumes, so let's focus on (a).
Out of the top of my head, the remote machine would need to:
- look for a common snapshot
- receive the btrfs send stream
- performing cleanup
(1) could be done by having a machine-friendly "snapsync list" subcommand that the local snapsync can parse.
(2) does not even necessarily have to be done through snapsync (local could remotely run the btrfs receive command)
(3) is a purely local operation (i.e. the current code, running on the remote host, should do the trick)
There is an argument to be made for (a): Not needing to install ruby etc on the backup server would be good. E.g. if I were to use my Synology NAS as a backup target I would only need to set up ssh and btrfs permissions to get it to work. I would be loathe to install ruby et al on what is essentially my most critical server.
I understand that this may complicate things, but lets just not dismiss (a) out of hand :-)
There is one more step, I think. Copy the snapper info.xml file(s). And the snapsync.config file?
I understand that this may complicate things, but lets just not dismiss (a) out of hand :-)
You have a very good point actually ... (a) is definitely an interesting use-case
There is one more step, I think. Copy the snapper info.xml file(s). And the snapsync.config file?
Yep.
So I'm using snapsync for more than a year now and I am really happy with it and experienced no issues with it (apart from a failing backup drive :D, but this is probably related to faulty hardware not to snapsync). Back to topic: I recently got a raspberrypi I also wanted to use as my backup server. Problem now: I would need to transfer the snapshot over SSH. So is there any progress on this? I would also volunteer to help implementing/implement this feature, if wanted. Only caveat with this would be that a) I don't know Ruby and b) I never got in touch with snapsync's codebase. Apart from that I would really love to get that feature in snapsync (and also I'm always open to learn new languages).
TL~DR: I would really like to be able to transfer my btrfs snapshots with snapsync over SHH and want to know what is the plan about this feature.
Because quite a bit of issues with btrfs, I'm actually not personally using (and therefore, developing) snapsync anymore.
I'd be happy to not let it die, though, so if you are willing to try, I can help you get in the codebase. I would be willing to give you rights on the github repo and to publish the gem as well if you'd like.
So, it's a pity you're not actually developing this, but I'm really glad and thankful you don't want to let snapsync die.
I'm not sure if I'm the right person for completely taking care of snapsync, because I never programmed in ruby ever before, although I'm open to it. Maybe I'll just see how well I can handle the backup over ssh feature and decide afterwards.
Back to the remote backup: Personally I agree with @DonOregano that not having to install snapsync on the remote server would be more comfortable from a users perspective, so I would like to implement your option b) from above (backup to a machine that does not have snapsync installed, but gives access to btrfs).
I'm not sure I got right how the backup process works, but anyway, here's my (very general) idea of how it could work (more or less what you've already outlined aboved):
- Another config option is added, containing the (user@)host name of the machine the backup should be made on
- when synchronizing a snapshot (and the remote option is given), we test if we can access the btrfs commands (probably i. e. testing if we have access to su/sudo and btrfs progs are installed) on the remote machine, test if the correct path is available, then make the backup
- the backup is made more or less the same way it is done now, with the only difference, that the commands concerning the receiving part of backing up (receiving the snapshot, finding a common snapshot, copying the info.xml file) are run on the remote machine via ssh
If I'm correct, this would mean I have to add the following things:
- Another config option
- establishing a ssh connection
- a modified version of the existing class(es) to run certain commands not on the local machine but via the ssh connection created earlier (a subclass of the btrfs class maybe?, not sure what would be appropriate here)
So, this is a lot of text I wrote here, I hope it's not too much (in the sense of annoying you). Again, I am really thankful you don't want to let snapsync die and are open to new ideas.
Hi @liketechnik ... sorry for the long delay. I'm pretty busy these days. I'm glad that snapsync is still useful to people.
The plan sounds reasonable.
It might be easier, in fine, to abstract the target out of LocalSync instead of having a completely separate logic between local and remote. In fine, a ssh target is really only a different way to run commands.
btrfs is currently used as a singleton module - a bad design choice, as usual. In addition to the "run btrfs remotely" functionality, you will also have to create a remote filesystem access shim for some actions (e.g. list existing snapshots, creating the partial marker file, ...).
- change Btrfs to become a proper class/object that gets injected in
LocalSync - provide a TargetFilesystem abstraction
or
- merge Btrfs and TargetFilesystem functionality into a Target class that would provide the whole necessary API for Sync to work
I've been using the net-ssh and net-ssh-sftp gems lately http://net-ssh.github.io/net-ssh/ recently to do remote stuff, with some success. Thought that might help.
Hi @doudou, don't worry about the delay, that's not a big problem for me.
I also came to the conclusion that it's probably a better solution, to abstract and create several local/remote versions of certain classes (including btrfs class and LocalSync).
Thanks for the hint on net-ssh, although I do not know yet how to add another library (is that how it is called in ruby?) to snapsync.
Not sure if this is the right place to ask because it does not necessarily have to do with this feature, but how did you test snapsync? I saw the test directory, but I have absolutely no clue how it could work. (Not to mention how to build the releases. Well, I had to start implementing instead of planing before thinking about testing and releasing, I guess :D)
.... once upon a time I was maintaining some unit tests. I think the ones that are there actually work. But they're testing the internal logic, not the btrfs layer, and definitely not the transfer access layer.
Very unfortunately, I did not spend the time setting up a proper testing environment - which would probably involve virtual machines (because btrfs).
Managing dependencies: snapsync uses bundler (as 99% of Ruby applications), check out https://bundler.io/. There's probably a bunch of tutorials out there.
Run your dev version bundler exec bin/snapsync
To run tests, just do bundler exec rake test
Once you're at the release ... let's do the first one together. It's really easy, but I have to give you access to the rubygems accout, I think it would be just better that we do the first one together live.