rsync
rsync copied to clipboard
WayneD
rsync -avzh hangs forever from 3.2.3 to 3.1.3 if file is bigger than 16GB
It is undoubtedly churning through such the huge quantity of checksum matches that it overwhelms the cpu. In the future I will probably change the sender to segment really large files into chunks so the transfer will be as efficient as if the large file had been broken-up into several more-manageable files without having to spend I/O on the breakup and reconstruction. i.e., it won't be able to use a checksum match for a late chunk in the file for an early chunk in the file, but it also won't bog down the checksum matching algorithm. The nice thing about this is that this change should be totally contained to the sender logic, where the receiving side doesn't need to know that the sender is limiting its matching into sub-file buckets.
