rsync icon indicating copy to clipboard operation
rsync copied to clipboard

Slow transfer after resume

Open GyrosGeier opened this issue 1 year ago • 0 comments

When resuming a transfer from a --partial file, the rsync algorithm is used for the entirety of the file. For the part already transferred, that is a reasonable speedup, but over a fast network, this slows down the rest of the transfer, and the sender is CPU bottlenecked.

To reproduce:

  1. create a large file with random data
  2. transfer the file
  3. truncate the destination file to a smaller size
  4. re-transfer the file

When I do this with a 50 GB file over a 1Gbps network, the beginning of the file is quickly checksummed, my machines manage this at about 500 MB/s, so this is not a generic performance issue. The remaining part of the file is then transferred at about 50 MB/s, with the rsync process taking 100% CPU. If I use the -W option to transfer the entire file from the beginning, I get the expected 108 MB/s, and very little CPU use.

GyrosGeier avatar Apr 19 '24 07:04 GyrosGeier