cloudpathlib
cloudpathlib copied to clipboard
S3Path: support extremely fast bucket-to-bucket transfers
We've experimented with this tool for fast bucket-to-bucket transfers. It would be cool if we could leverage the logic here to make an incredibly fast bucket-to-bucket transfer: https://github.com/generalui/s3p/
Here's a blog post with more deets: https://medium.com/@shanebdavis/s3p-massively-parallel-s3-copying-9a9e466d0d74
I think the core pieces are:
-
async
for all of the requests - use
prefix
and assumptions about list results in alphabetical order to split up the parallel "list" calls search space? (or something like that) - get a massive number of requests goin' at once.
(This may also be out of scope for the package, just throwing it out there as useful for our purposes)