rsync
rsync copied to clipboard
Slow transfer after resume
When resuming a transfer from a --partial file, the rsync algorithm is used for the entirety of the file. For the part already transferred, that is a reasonable speedup, but over a fast network, this slows down the rest of the transfer, and the sender is CPU bottlenecked.
To reproduce:
- create a large file with random data
- transfer the file
- truncate the destination file to a smaller size
- re-transfer the file
When I do this with a 50 GB file over a 1Gbps network, the beginning of the file is quickly checksummed, my machines manage this at about 500 MB/s, so this is not a generic performance issue. The remaining part of the file is then transferred at about 50 MB/s, with the rsync process taking 100% CPU. If I use the -W option to transfer the entire file from the beginning, I get the expected 108 MB/s, and very little CPU use.