s3cp icon indicating copy to clipboard operation
s3cp copied to clipboard

Always be downloading

Open balshor opened this issue 10 years ago • 1 comments

When downloading multiple files (s3cp -r), s3cp performs a checksum after each file before moving on to the next. It would be more efficient to start the download of the next file in parallel with checksum on the previous file as the checksum is (presumably) CPU bound while the download is IO bound. Instead of immediately retrying, files that need to be re-downloaded due to corruption can be added to the end of the download queue for retrying later.

balshor avatar Dec 06 '13 00:12 balshor