boto_rsync
boto_rsync copied to clipboard
speed up 15x with parallel transfers?
Not sure if this has been looked into already, but the speed of transfers seems to be vastly improved by threads/processes running in parallel.
My experience with is is using dreamhost, it's possible the official s3 is different.
I noticed a 5x improvement in speed in calling boto-rsync on each of 5 large directories, rather than on the single parent directory. I assume an internal system for using multiple upload threads/queues would see similar improvements.