server
server copied to clipboard
nextcloud
[Bug]: Exception using GCS as primary storage: Rewriting objects created via Multipart Upload is not implemented yet
⚠️ This issue respects the following points: ⚠️
- [X] This is a bug, not a question or a configuration/webserver/proxy issue.
- [X] This issue is not already reported on Github (I've searched it).
- [X] Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
- [X] Nextcloud Server is running on 64bit capable CPU, PHP and OS.
- [X] I agree to follow Nextcloud's Code of Conduct.
Bug description
Exception when upload large file to a shared folder (of other user) using GCS as primary storage. This only happens with shared folder, but can upload successfully with my own folder.
Steps to reproduce
- Setup next cloud and using GCS as a S3 primary storage
- User A create a folder and share this folder to user B
- User B login + upload a file with 200 MB size
- Server return this errror
Error executing "CopyObject" on "https://honganh-production-nextcloud.storage.googleapis.com/urn%3Aoid%3A1664"; AWS HTTP error: Client error: PUT https://honganh-production-nextcloud.storage.googleapis.com/urn%3Aoid%3A1664 resulted in a 400 Bad Request response:
InvalidArgumentInvalidArgument (client): Invalid argument. - <Error><Code>InvalidArgument</Code><Message>Invalid argument.</Message><Details>Rewriting objects created via Multipart Upload is not implemented yet. As a workaround, you can use compose to overwrite the object (by specifying honganh-production-nextcloud/urn:oid:1663 as both the source and output of compose) prior to rewrite.</Details></Error>
Expected behavior
Upload file succes
Installation method
Community Docker image
Nextcloud Server version
26.0.1
Operating system
Debian/Ubuntu
PHP engine version
PHP 8.0
Web server
Other
Database engine version
None
Is this bug present after an update or on a fresh install?
None
Are you using the Nextcloud Server Encryption module?
None
What user-backends are you using?
- [X] Default user-backend (database)
- [ ] LDAP/ Active Directory
- [ ] SSO - SAML
- [ ] Other
Configuration report
No response
List of activated Apps
Default apps
Nextcloud Signing status
No response
Nextcloud Logs
No response
Additional info
I think it's relead to this consideration https://cloud.google.com/storage/docs/multipart-uploads#considerations. An object uploaded using this method cannot be copied or rewritten, unless you compose the object prior to the copy or rewrite.
Hi @thanhpd56:
I can't reproduce this issue on AWS S3. I'd like to take a closer look at at it on GCS's S3 compatibility mode.
Can you provide your NC configuration as requested in the submission form? (without your passwords/keys obviously)
Also, can you provide the entire stack trace associated with the log entry rather than just a snippet? Easiest way to grab this would be to go to Admin->Logging and move your cursor over to the right of the entry in question then selection Copy raw.
When including a stack trace/configs/etc, make sure to indicate in the GitHub editor it is:
preformatted / code
...so that it doesn't get reformatted or we'll be unable to review it. Thanks.
A brief follow-up. I believe you were on the right track about the culprit:
https://cloud.google.com/storage/docs/release-notes#October_12_2021 https://cloud.google.com/storage/docs/multipart-uploads#considerations
Specifically, even though GCS has an S3 "compatibility" layer, it's not actually entirely up to spec with AWS S3 (the compose stuff is GCS specific and, unfortunately, kind of important).
In some situations, even when that error you're seeing appears, operations still work (because of some fallback behavior that helps in some cases in NC I think). But in other cases it's a true hard error. For example, I can trigger the error doing a copy while utilizing GCS, but the file does ultimately copy. But I can also trigger the error doing a move and the file never moves. (For the record, this testing was with External Storage not Primary Storage so there are some differences).
NC utilizes the official aws-sdk-php for S3 access so I'm not really sure we're going to be able to do much about this directly.
Google has evolved their S3 support over time. As a customer it might be worth lodging a support case with them to see when they intend to fix this issue...
Other S3 compatible providers like Backblaze B2 and off-the-shelf solutions like MinIO do not have this problem.
I believe this is the tracker on the Google side for this: https://issuetracker.google.com/issues/229020040
Go vote that you're impacted by it.
Bottom line: This is not an NC issue, but a GCS S3 compatibility issue.
This looks finally fixed on Google's end:
https://cloud.google.com/storage/docs/release-notes#June_23_2023
Objects created using XML API multipart uploads can now be copied and rewritten normally. Previously, you had to perform an object composition on such objects before the output could be copied or rewritten.
This looks finally fixed on Google's end:
https://cloud.google.com/storage/docs/release-notes#June_23_2023
Objects created using XML API multipart uploads can now be copied and rewritten normally. Previously, you had to perform an object composition on such objects before the output could be copied or rewritten.
Thanks for your info. Infact I switched to S3 as main storage, and using Terraform for multi cloud management. However, multiple billings still is the problem. :v