s3 storage plugin upload concurrency

[GeorgeShao]

📚 Question

From what I see, this library's s3 storage plugin uses the aiobotocore library's put_object function, which sends a single PutObject request, rather than using multipart uploads for greater upload concurrency.

I created a patch of torchsnapshot that uses the aioboto3 library's upload_fileobj function. In my tests, it significantly increased write performance for large files.

Is there a reason torchsnapshot's s3 storage plugin doesn't support multipart uploads?

I see that DCP has superseded torchsnapshot. Is there an easy migration path for codebases still using torchsnapshot?