Figure out a scalable rsync solution

[humphd]

One killer feature to consider is doing rsync, within a filesystem (e.g., mirroring dir A/ to dir B/) and later between filesystems.

One fast path to this would be to adopt/adapt anchor (rsync in node.js):

https://github.com/ttezel/anchor/blob/master/lib/anchor.js

[humphd]

@pbouianov has started an implementation of this over at https://github.com/pbouianov/filer/blob/rsync/src/rsync.js. Some things to do with this code:

• Get rid of RSync, and just return the rsync function from require
• Accept the size property as an option to the rsync function, and assume a sane default block size. This will be something that we might want to change later when we fix issue #10
• Decide where we want to store the temp cache for file operations in rsync (e.g., per shell, one global?)
• Probably keep the rsync code in separate file, and attach to the Shell.prototype using require
• Add comments indicating where you borrowed/stole code, and what license was used (link to github, etc).

This wants TESTS. Also, it would be good to implement issue #121 in terms of sh.rsync.

[humphd]

This remains a dream of mine. Today I found this paper:

http://www.cs.yale.edu/homes/zhai-ennan/fast18-final70.pdf

And the full implementations of various algorithms from the paper in JS:

https://github.com/WebDeltaSync/WebRsync, https://github.com/WebDeltaSync/WebR2sync_plus (see their other repos as well). It would be interesting to get this working with Filer.

[humphd]

Summary blog post about the work: https://blog.acolyer.org/2018/03/02/towards-web-based-delta-synchronization-for-cloud-storage-systems/

[humphd]

https://github.com/zbackup/zbackup from #348. Looks like it's dead, but interesting.

[humphd]

http://ori.scs.stanford.edu/# from #343. Also looks cool. Also looks dead :(

[humphd]

https://github.com/syncthing/syncthing from #307. Looks cool, still active.