boto_rsync icon indicating copy to clipboard operation
boto_rsync copied to clipboard

speed up 15x with parallel transfers?

Open inspire22 opened this issue 11 years ago • 0 comments

Not sure if this has been looked into already, but the speed of transfers seems to be vastly improved by threads/processes running in parallel.

My experience with is is using dreamhost, it's possible the official s3 is different.

I noticed a 5x improvement in speed in calling boto-rsync on each of 5 large directories, rather than on the single parent directory. I assume an internal system for using multiple upload threads/queues would see similar improvements.

inspire22 avatar Jan 07 '14 17:01 inspire22