nextcloud
New S3 Configuration breaks prior S3 configuration and is missing uploadPartSize
Describe your Issue
I've used the S3 integration before "manually" via configs.s3.config.php, now the helm chart upgrade (v5.4.0) fails because it's already defined
Logs and Errors
Helm upgrade failed: error while running post render on files: map[string]interface {}(nil): yaml: unmarshal errors: line 168: mapping key "s3.config.php" already defined at line 23
Describe your Environment
-
Kubernetes distribution: k8s
-
Helm Version (or App that manages helm):
-
Helm Chart Version: 5.4.0
-
values.yaml:
configs:
s3.config.php: |-
<?php
$CONFIG = array (
'objectstore' => array(
'class' => '\\OC\\Files\\ObjectStore\\S3',
'arguments' => array(
'hostname' => 's3.eu-central-2.wasabisys.com',
'bucket' => '...',
'autocreate' => false,
'key' => '...',
'secret' => '...',
'region' => 'eu-central-2',
'use_ssl' => true,
'use_path_style' => false,
'uploadPartSize' => '2097152000'
)
)
);
Additional context, if any
- It would be nice if such a change would be marked as a breaking change next time
- The chart should respect the manually set configuration
- The new options are missing
uploadPartSize, which is required to be set there for some reason on S3 storage, so I can't even switch to the new configuration style
Sorry for this breaking change :/ I think your config should continue to work fine by just renaming the config file.
Could you open a PR for the uploadPartSize so that it is present and allows you to switch to the new configs?
@provokateurin thanks for the quick response, I can try to open a PR but I'm no expert in helm charts and I can't test the change locally at the moment.
🙇 sorry about that, @acidicX!
I actually submitted https://github.com/nextcloud/docker/pull/2271 upstream for these changes, but I'll submit one here as well now!
In the meantime, you should be able to disable the default config being deployed by setting nextcloud.defaultConfigs.s3.config.php=false, or via values.yaml:
nextcloud:
defaultConfigs:
s3.config.php: false
Can you confirm if this works? Because if that doesn't work either, we should also try to fix that. Sorry again for the inconvenience regardless 🙏
Sorry for the rapid updates, but if #614 gets merged, it will close this Issue automatically. That doesn't mean we can't still help you if other things are still broken here though, so please just let us know if that's the case and we can reopen this issue for you if it gets closed 🙏
Update: We're going to wait till nextcloud/docker reviews/merges our upstream PR, https://github.com/nextcloud/docker/pull/2271 before merging #614
@jessebot thanks for the fast reaction, I can confirm that the workaround (nextcloud.defaultConfigs.s3.config.php=false) works for me :) I've been able to upgrade and everything seems to work fine.
Hello @jessebot , @acidicX ! I got the same trouble with already defined mapping key for s3.config.php file. I used:
nextcloud:
defaultConfigs:
s3.config.php: false
and it does not work for me. I try for 5.4.0 and 5.5.2 chart versions.
I was able to fix it with renaming my s3 config file to:
configs:
s33.config.php:
Unfortunately it still needs to be fix. :(
Essential part of my helm release file:
defaultConfigs:
s3.config.php: false
swift.config.php: false
objectStore:
s3:
enabled: true
ssl: true
port: ***
region: ***
usePathStyle: true
autoCreate: true
existingSecret: nextcloud-values
secretKeys:
host: hostname
accessKey: key
secretKey: secret
bucket: bucket
@MaksonHerson I'm sorry about that. Could you check and see if s3.config.php is actually already in your config directory? Perhaps you just need to delete it? 🤔 Either way, let me know and I'm happy to keep troubleshooting it with you.
@jessebot this now does not work after #464 with sse_c_key specified or not. I tried both:
level":3,"time":"2024-10-06T23:05:35+00:00","remoteAddr":"10.42.7.135","user":"jstewart","app":"objectstore","method":"GET","url":"/avatar/jstewart/64/dark?v=17282
55935126","message":"{\\"Exception\\":\\"Exception\\",\\"Message\\":\\"Creation of bucket \\\\\\"nextcloud\\\\\\" failed. Error executing \\\\\\"CreateBucket\\\\\\
" on \\\\\\"https://hl15-1.home.thecrimsontint.com:19000/nextcloud/\\\\\\"; AWS HTTP error: Client error: `PUT https://hl15-1.home.thecrimsontint.com:19000/nextclo
ud/` resulted in a `403 Forbidden` response:\\\\n<?xml version=\\\\\\"1.0\\\\\\" encoding=\\\\\\"UTF-8\\\\\\"?>\\\\n<Error><Code>SignatureDoesNotMatch</Code><Messa
ge>The request signature we calcul (truncated...)\\\\n SignatureDoesNotMatch (client): The request signature we calculated does not match the signature you provide
d. Check your key and signing method. - <?xml version=\\\\\\"1.0\\\\\\" encoding=\\\\\\"UTF-8\\\\\\"?>\\\\n<Error><Code>SignatureDoesNotMatch</Code><Message>The re
quest signature we calculated does not match the signature you provided. Check your key and signing method.</Message><BucketName>nextcloud</BucketName><Resource>/n
extcloud/</Resource><RequestId>17FBFF9DE0874B12</RequestId><HostId>470a0afc0f6fdbebb6a8e14ece29122784c0656e19563387135a7c8e02feb163</HostId></Error>\\",\\"Code\\":
0,\\"Trace\\":[{\\"file\\":\\"/var/www/html/lib/private/Files/ObjectStore/S3ObjectTrait.php\\",\\"line\\":90,\\"function\\":\\"getConnection\\",\\"class\\":\\"OC\\
\\\\\\Files\\\\\\\\ObjectStore\\\\\\\\S3\\",\\"type\\":\\"->\\",\\"args\\":[]},{\\"file\\":\\"/var/www/html/lib/private/Files/ObjectStore/S3ObjectTrait.php\\",\\"l
ine\\":152,\\"function\\":\\"writeSingle\\",\\"class\\":\\"OC\\\\\\\\Files\\\\\\\\ObjectStore\\\\\\\\S3\\",\\"type\\":\\"->\\",\\"args\\":[\\"urn:oid:20\\",[\\"Guz
zleHttp\\\\\\\\Psr7\\\\\\\\Stream\\"],\\"image/png\\"]},{\\"file\\":\\"/var/www/html/lib/private/Files/ObjectStore/ObjectStoreStorage.php\\",\\"line\\":520,\\"func
tion\\":\\"writeObject\\",\\"class\\":\\"OC\\\\\\\\Files\\\\\\\\ObjectStore\\\\\\\\S3\\",\\"type\\":\\"->\\",\\"args\\":[\\"urn:oid:20\\",null,\\"image/png\\"]},{\
\"file\\":\\"/var/www/html/lib/private/Files/ObjectStore/ObjectStoreStorage.php\\",\\"line\\":446,\\"function\\":\\"writeStream\\",\\"class\\":\\"OC\\\\\\\\Files\\
\\\\\\ObjectStore\\\\\\\\ObjectStoreStorage\\",\\"type\\":\\"->\\",\\"args\\":[\\"appdata_ocy4keitmxg4/avatar/jstewart/avatar-dark.png\\",null,9351]},{\\"file\\":\
\"/var/www/html/lib/private/Files/ObjectStore/ObjectStoreStorage.php\\",\\"line\\":354,\\"function\\":\\"writeBack\\",\\"class\\":\\"OC\\\\\\\\Files\\\\\\\\ObjectS
tore\\\\\\\\ObjectStoreStorage\\",\\"type\\":\\"->\\",\\"args\\":[\\"/tmp/oc_tmp_5ZfU10-.png\\",\\"appdata_ocy4keitmxg4/avatar/jstewart/avatar-dark.png\\"]},{\\"fu
nction\\":\\"OC\\\\\\\\Files\\\\\\\\ObjectStore\\\\\\\\{closure}\\",\\"class\\":\\"OC\\\\\\\\Files\\\\\\\\ObjectStore\\\\\\\\ObjectStoreStorage\\",\\"type\\":\\"->
\\",\\"args\\":[\\"*** sensitive parameters replaced ***\\"]},{\\"file\\":\\"/var/www/html/3rdparty/icewind/streams/src/CallbackWrapper.php\\",\\"line\\":119,\\"fu
nction\\":\\"call_user_func\\",\\"args\\":[[\\"Closure\\"]]},{\\"function\\":\\"stream_close\\",\\"class\\":\\"Icewind\\\\\\\\Streams\\\\\\\\CallbackWrapper\\",\\"
type\\":\\"->\\",\\"args\\":[\\"*** sensitive parameters replaced ***\\"]},{\\"file\\":\\"/var/www/html/lib/private/Files/ObjectStore/ObjectStoreStorage.php\\",\\"
line\\":470,\\"function\\":\\"fclose\\",\\"args\\":[\\"*** sensitive parameters replaced ***\\"]},{\\"file\\":\\"/var/www/html/lib/private/Files/Storage/Wrapper/Wr
apper.php\\",\\"line\\":240,\\"function\\":\\"file_put_contents\\",\\"class\\":\\"OC\\\\\\\\Files\\\\\\\\ObjectStore\\\\\\\\ObjectStoreStorage\\",\\"type\\":\\"->\
\",\\"args\\":[\\"appdata_ocy4keitmxg4/avatar/jstewart/avatar-dark.png\\",null]},{\\"file\\":\\"/var/www/html/lib/private/Files/View.php\\",\\"line\\":1134,\\"func
tion\\":\\"file_put_contents\\",\\"class\\":\\"OC\\\\\\\\Files\\\\\\\\Storage\\\\\\\\Wrapper\\\\\\\\Wrapper\\",\\"type\\":\\"->\\",\\"args\\":[\\"appdata_ocy4keitm
xg4/avatar/jstewart/avatar-dark.png\\",null]},{\\"file\\":\\"/var/www/html/lib/private/Files/View.php\\",\\"line\\":644,\\"function\\":\\"basicOperation\\",\\"clas
s\\":\\"OC\\\\\\\\Files\\\\\\\\View\\",\\"type\\":\\"->\\",\\"args\\":[\\"file_put_contents\\",\\"/appdata_ocy4keitmxg4/avatar/jstewart/avatar-dark.png\\",[\\"crea
te\\",\\"write\\"],null]},{\\"file\\":\\"/var/www/html/lib/private/Files/Node/Folder.php\\",\\"line\\":167,\\"function\\":\\"file_put_contents\\",\\"class\\":\\"OC
\\\\\\\\Files\\\\\\\\View\\",\\"type\\":\\"->\\",\\"args\\":[\\"/appdata_ocy4keitmxg4/avatar/jstewart/avatar-dark.png\\",null]},{\\"file\\":\\"/var/www/html/lib/pr
ivate/Files/SimpleFS/NewSimpleFile.php\\",\\"line\\":103,\\"function\\":\\"newFile\\",\\"class\\":\\"OC\\\\\\\\Files\\\\\\\\Node\\\\\\\\Folder\\",\\"type\\":\\"->\\",\\"args\\":[\\"avatar-dark.png\\",null]},{\\"file\\":\\"/var/www/html/lib/private/Avatar/UserAvatar.php\\",\\"line\\":208,\\"function\\":\\"putContent\\",\\"cla
ss\\":\\"OC\\\\\\\\Files\\\\\\\\SimpleFS\\\\\\\\NewSimpleFile\\",\\"type\\":\\"->\\",\\"args\\":[null]},{\\"file\\":\\"/var/www/html/core/Controller/AvatarControll
er.php\\",\\"line\\":84,\\"function\\":\\"getFile\\",\\"class\\":\\"OC\\\\\\\\Avatar\\\\\\\\UserAvatar\\",\\"type\\":\\"->\\",\\"args\\":[64,true]},{\\"file\\":\\"
/var/www/html/lib/private/AppFramework/Http/Dispatcher.php\\",\\"line\\":208,\\"function\\":\\"getAvatarDark\\",\\"class\\":\\"OC\\\\\\\\Core\\\\\\\\Controller\\\\
\\\\AvatarController\\",\\"type\\":\\"->\\",\\"args\\":[\\"jstewart\\",64,false]},{\\"file\\":\\"/var/www/html/lib/private/AppFramework/Http/Dispatcher.php\\",\\"l
ine\\":114,\\"function\\":\\"executeController\\",\\"class\\":\\"OC\\\\\\\\AppFramework\\\\\\\\Http\\\\\\\\Dispatcher\\",\\"type\\":\\"->\\",\\"args\\":[[\\"OC\\\\
\\\\Core\\\\\\\\Controller\\\\\\\\AvatarController\\"],\\"getAvatarDark\\"]},{\\"file\\":\\"/var/www/html/lib/private/AppFramework/App.php\\",\\"line\\":161,\\"fun
ction\\":\\"dispatch\\",\\"class\\":\\"OC\\\\\\\\AppFramework\\\\\\\\Http\\\\\\\\Dispatcher\\",\\"type\\":\\"->\\",\\"args\\":[[\\"OC\\\\\\\\Core\\\\\\\\Controller\\\\\\\\AvatarController\\"],\\"getAvatarDark\\"]},{\\"file\\":\\"/var/www/html/lib/private/Route/Router.php\\",\\"line\\":302,\\"function\\":\\"main\\",\\"class\\
":\\"OC\\\\\\\\AppFramework\\\\\\\\App\\",\\"type\\":\\"::\\",\\"args\\":[\\"OC\\\\\\\\Core\\\\\\\\Controller\\\\\\\\AvatarController\\",\\"getAvatarDark\\",[\\"OC\\\\\\\\AppFramework\\\\\\\\DependencyInjection\\\\\\\\DIContainer\\"],[\\"jstewart\\",\\"64\\",\\"core.avatar.getavatardark\\"]]},{\\"file\\":\\"/var/www/html/lib
/base.php\\",\\"line\\":1001,\\"function\\":\\"match\\",\\"class\\":\\"OC\\\\\\\\Route\\\\\\\\Router\\",\\"type\\":\\"->\\",\\"args\\":[\\"/avatar/jstewart/64/dark\\"]},{\\"file\\":\\"/var/www/html/index.php\\",\\"line\\":24,\\"function\\":\\"handleRequest\\",\\"class\\":\\"OC\\",\\"type\\":\\"::\\",\\"args\\":[]}],\\"File\\
":\\"/var/www/html/lib/private/Files/ObjectStore/S3ConnectionTrait.php\\",\\"Line\\":145,\\"message\\":\\"Could not create object urn:oid:20 for appdata_ocy4keitmx
g4/avatar/jstewart/avatar-dark.png\\",\\"exception\\":{},\\"CustomMessage\\":\\"Could not create object urn:oid:20 for appdata_ocy4keitmxg4/avatar/jstewart/avatar-
dark.png\\"}","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36","version":"30.0.0.14"}
nextcloud 10.42.7.135 - - [06/Oct/2024:23:05:35 +0000] "GET /avatar/jstewart/64/dark?v=1728255935126 HTTP/1.1" 404 753 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x6
4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"```
hi, I found a change of copySizeLimit to a value larger than default has no effect.
@acidicX @jstewart612 Have you ever met the issue that user cannot upload a file larger than 5GB with enabling s3 backend. great thanks!
I've been having issue with uploading file larger than ~2GB to a Ceph RGW with S3 protocol. It will fail with SSL error or 502 bad gateway. Using s3cmd with a pod in a same Kubernetes cluster works fine. Haven't solve the issue yet.
