s3s3mirror icon indicating copy to clipboard operation
s3s3mirror copied to clipboard
verified publisher icon cobbzilla

Multipart upload does not scale

Open alexandruast opened this issue 9 years ago • 1 comments

The multipart upload does not scale well. For example, on a bucket with 120k objects, first 500GB and 119k+ objects get copied in 30 minutes. There are less than 20 files left with sizes over 5GB. Those take forever to copy, because the chunks are not processed in parallel (almost 60 minutes for remaining 100GB)

alexandruast avatar Mar 10 '16 11:03 alexandruast

This is one of those odd issues that I am marking both "enhancement" and "bug" -- I would welcome a pull request that uploads chunks in parallel.

cobbzilla avatar Mar 11 '16 03:03 cobbzilla