googleapis
storage: Support parallel composite uploads for faster transfers
Is your feature request related to a problem? Please describe.
gsutil supports parallel composite uploads, but the Google Cloud Storage SDKs do not.
For large files, transfers to Google Cloud Storage could be significantly faster if chunks of data are uploaded simultaneously. The Amazon SDKs natively support multi-threaded, multipart uploads with Golang, Ruby, and Python libraries.
Describe the solution you'd like
gsutil implements this in Python.
Ideally there would be a simple boolean parameter, such as ParallelCompositeUpload, that would activate this mode in the bucket Write.
Describe alternatives you've considered
This could be implemented in the application layer, but it really belongs in the SDK.
Additional context Add any other context or screenshots about the feature request here.
I'd say this is comparable to #1672. It's a very reasonable feature request, but would entail more involved client-side logic than we usually include in this library, which is a fairly thin client. I'll raise internally whether this can be prioritized.
@tritone Can you please ask internally about the similar request to add this in google-cloud-ruby as well?
googleapis/google-cloud-ruby#8147
@quartzmo I don't think we are prioritizing this in the next quarter at least. @frankyn may have more context for Ruby specifically.
We are not yet prioritizing this work across languages. It's still in our backlog, and we are working our way back to it.
We do, but it's competing with other work items don't have ETA yet. Thanks for your patience.
Just wanted to provide a long-delayed update here. We're planning on implementing parallel upload and download features in the newly-created subpackage transfermanager. We did the initial merge for downloads this week, more download features are coming later this month, and uploads should be coming in the second half of the year most likely. Feel free to take a look here: https://pkg.go.dev/cloud.google.com/go/[email protected]/transfermanager ; we are open to hearing feedback.
@tritone great news, hope you will update this issue when Uploader will implements
Hi, @tritone, I am rather interested in this feature, so since it's been few months since last update just wanted to as, maybe you know if it is currently scheduled or some ETA is assigned? Thanks.
Hi @tritone, we're also very interested in Go support for this feature! Just wanted to ask when you think it might be ready for production use? Thanks!
Base on https://pkg.go.dev/cloud.google.com/go/storage/transfermanager and https://github.com/googleapis/google-cloud-go/tree/storage/v1.51.0/storage/transfermanager I guess it is already released and ready for use?
