cloudpathlib icon indicating copy to clipboard operation
cloudpathlib copied to clipboard

S3Path: support extremely fast bucket-to-bucket transfers

Open pjbull opened this issue 4 years ago • 0 comments

We've experimented with this tool for fast bucket-to-bucket transfers. It would be cool if we could leverage the logic here to make an incredibly fast bucket-to-bucket transfer: https://github.com/generalui/s3p/

Here's a blog post with more deets: https://medium.com/@shanebdavis/s3p-massively-parallel-s3-copying-9a9e466d0d74

I think the core pieces are:

  • async for all of the requests
  • use prefix and assumptions about list results in alphabetical order to split up the parallel "list" calls search space? (or something like that)
  • get a massive number of requests goin' at once.

(This may also be out of scope for the package, just throwing it out there as useful for our purposes)

pjbull avatar Sep 03 '20 04:09 pjbull