rsinc icon indicating copy to clipboard operation
rsinc copied to clipboard

rsinc must wait for a LONG time for rclone to hash large (severl GB) files

Open ConorWilliams opened this issue 6 years ago • 1 comments

Rsinc requires an 'ID' for each file which is currently a concatenation of a files hash and its size. This must be unique for each file and independant of its name/path to allow file moves/renames to be tracked. Unfortuntly computing this hash for large files (e.g films) seems to take a huge amount of time (up to minutes per file). I welcome any suggestions to fix this problem. Ideally some way to cache the hashes and only recompute them if the file is modified?

ConorWilliams avatar Oct 01 '19 12:10 ConorWilliams

I have mitigated this problem by splitting rsinc's hashing out of the rclone lsjson command and into the rclone hashsum command. This has the benafit of only requiring one hash per file to be computed (instead of 5+ that rclone lsjson does!) and using only the fastes avaliable hash, resulting in a significant speed up.

ConorWilliams avatar Oct 13 '19 02:10 ConorWilliams