Support multipart-uploads (MPU) in the client library

[coryan]

This is a new feature in GCS:

https://cloud.google.com/storage/docs/xml-api/post-object-multipart https://cloud.google.com/storage/docs/xml-api/put-object-multipart https://cloud.google.com/storage/docs/xml-api/post-object-complete https://cloud.google.com/storage/docs/xml-api/delete-multipart https://cloud.google.com/storage/docs/xml-api/get-bucket-uploads

It requires parsing some XML, but at least the formats are fixed.

[dulleh]

Any idea of an ETA for this feature?

We currently use the compose API and it has limitations and issues that we could try to work around but would prefer to wait for this feature if it's coming soon.

Key benefits of this API relevant to users of this SDK:

1. No need to make delete calls for the parts client-side so no development effort on writing a fast deletion process. It's naturally faster and cheaper.
2. Parts are not subject to early deletion fees (when nothing goes wrong), which means we can make this the default upload method. With the current API we have to maintain two upload methods.
3. Can list ongoing uploads so can do resumable uploads that are supported by server-side behaviour.
4. S3-like API makes it easier to develop cross-platform apps.

[coryan]

Thanks for reaching out. Unfortunately I do not think we will be able to work on this during 2021-Q4.

[devjgm]

Still want to do this, but no time yet.

[coryan]

Strike 2. We will want to do it, but no time yet.

[tmatsuo]

If someone really needs this feature, consider hiring me as a temporary freelance engineer (my github bio has contact).

Tasks included:

• Propose a design to the google-cloud-cpp team at google and get approved
• Send PRs for implementation and get merged

Background: I was assigned to this project starting from 2023. It was still in early prototype and design phase. @coryan and I agreed to have a simple XML parser in google-cloud-cpp so I started implementing it (#10558, #10564, #10568, #10608). The overall design was still in early stage, but @coryan and I discussed several times and I have a rough idea of the overall design.

Then unfortunately I got affected by the recent layoff 😥 I'll need to spend my time for surviving this situation, so I can't work on this project anymore unless someone hires me 😅

@coryan Feel free to

• delete this post if you feel it's spammy
• endorse me if you feel this proposal is reasonable (and you still like me)

[coryan]

I do not think we will have time to work on this during 2023-H2.

[coryan]

This is not scheduled for 2023, maybe in 2024.

[coryan]

We have decided not to work on this. We believe Parallel Composite Uploads cover most cases, and in C++ we already implement PCUs via ParallelUploadFile()