Investigate sparse & large file handling

[JohnStrunk]

Describe the feature you'd like to have. Investigate how the various movers handle both sparse and large files.

What is the value to the end user? (why is it a priority?) VM images, in particular, can be rather large. We should characterize what performance we expect (and optimize where we can). These images may also be sparse, and detecting/preserving that can also be a big benefit.

How will we know we have a good solution? (acceptance criteria)

• Understand what performance we should expect w/ large files.
  • Are small changes in the file handled efficiently?
  • Are there any optimizations that would make it better?
• Understand how sparse files are handled.
  • Is sparseness preserved?

Additional context Rsync & sparse: https://gergap.wordpress.com/2013/08/10/rsync-and-sparse-files/ Related to backube/scribe#122 Prompted by @mykaul

[mykaul]

You should look at what has been implemented in oVirt's image-io (see https://github.com/oVirt/ovirt-imageio/blob/master/examples/sparse-stream for example) to optimize sparse image copy.

[moonlight16]

It would be interesting to compare a NFS-based solution vs rsync. NFS is supposed to handle large file transfers better. Since rsync is build on top of SSH, I wonder if this is the bottleneck.