Sync multiple files in parallel

[hwittenborn]

Currently we sync files one at a time, which isn't the best choice for performance. We could sync multiple files in parallel across multiple threads (or by just using async), which could allow for much quicker file syncs.

Potentially fixes #64.

[theCalcaholic]

It would probably make sense to have different limits for generating checksums/checking for changes and transfers though - e.g. a maximum number of files to transfer per remote (a low number, e.g. 4), because here the network will probably be the bottleneck and a maximum number of files to check for changes based on the number of available CPU cores.

Assuming, of course, that this makes sense for your/rclone's architecture

[Gotterbild]

@theCalcaholic I don't think the network will be the bottleneck. I.e. in Proton Drive I have a folder with about 200 small files, ~70 MB total. It synced for about 4 hours, which shows that network is definitely not the bottleneck. My network is not that bad since I watch YouTube 1080p quality without any speed issues. :slightly_smiling_face:

Also, I tried to sync with rclone directly with the default 8 files in parallel and it synced in about 20 minutes.

Adding the ability to set --checkers for the Rclone in the UI will greatly increase the performance.

[theCalcaholic]

@Gotterbild I think you misunderstood me.

I am saying that are some point you don't get any speedup by transferring more files in parallel because you will just transfer more files slower (once you reach the maximum available bandwidth).