[Bug]: "Message":"Missing metadata for chunked upload" and for file upload > 10 MB

[sjash]

⚠️ This issue respects the following points: ⚠️

• [X] This is a bug, not a question or a configuration/webserver/proxy issue.
• [X] This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
• [X] Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• [X] I agree to follow Nextcloud's Code of Conduct.

Bug description

My Nextcloud instance version is 26.0.2 with mariadb and s3 as primary storage. After all configuration, I'm unable tyo upload large file of size 10MB and higher. The same configuration is working perfectly for version 25.

Steps to reproduce

1. Upload a file of size > 10MB.
2. Got error in nextcloud. log file as

{"reqId":"ZIlFLoMmYtUHgHCLRcxebgAAzQM","level":0,"time":"2023-06-14T10:12:22+05:30","remoteAddr":"10.20.4.25","user":"ecadmin","app":"webdav","method":"PUT","url":"/remote.php/dav/uploads/ecadmin/web-file-upload-0017df3152a12b2989c7740d0c8fe876-1686717741220/1","message":"Missing metadata for chunked upload","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36

"/web/service/apps/dav/appinfo/v2/remote.php"],"function":"require_once"}],"File":"/web/service/apps/dav/lib/Connector/Sabre/Directory.php","Line":227,"message":"File with name /Test/AM-eval-6.5.1.zip could not be located","exception":{},"CustomMessage":"File with name /Test/AM-eval-6.5.1.zip could not be located"}}

Expected behavior

The issue exist in version 26 and above where s3 primary storage is selected. The same configuration is working perfectly for version 25.

Installation method

Community Manual installation with Archive

Nextcloud Server version

26

Operating system

RHEL/CentOS

PHP engine version

PHP 8.2

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Fresh Nextcloud Server install

Are you using the Nextcloud Server Encryption module?

None

What user-backends are you using?

• [X] Default user-backend (database)
• [ ] LDAP/ Active Directory
• [ ] SSO - SAML
• [ ] Other

Configuration report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "mccloud.example.in"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "26.0.2.1",
        "overwrite.cli.url": "https:\/\/mccloud.example.in",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "ldapProviderFactory": "OCA\\User_LDAP\\LDAPProviderFactory",
        "objectstore": {
            "class": "\\OC\\Files\\ObjectStore\\S3",
            "arguments": {
                "bucket": "mccloud1",
                "autocreate": true,
                "key": "***REMOVED SENSITIVE VALUE***",
                "secret": "***REMOVED SENSITIVE VALUE***",
                "hostname": "s3-pdc.example.in",
                "port": 443,
                "use_ssl": true,
                "region": "optional",
                "use_path_style": true
            }
        },
        "maintenance": false,
        "bulkupload.enabled": "true",
        "filelocking.enabled": false,
        "knowledgebaseenabled": false,
        "logtimezone": "Asia\/Kolkata",
        "log_rotate_size": 52428800,
        "loglevel": 0,
        "lost_password_link": "https:\/\/adauth.example.in\/forget-password",
        "app_install_overwrite": []
    }
}

List of activated Apps

Enabled:
  - activity: 2.18.0
  - admin_audit: 1.16.0
  - cloud_federation_api: 1.9.0
  - comments: 1.16.0
  - contactsinteraction: 1.7.0
  - dashboard: 7.6.0
  - dav: 1.25.0
  - federatedfilesharing: 1.16.0
  - federation: 1.16.0
  - files: 1.21.1
  - files_external: 1.18.0
  - files_pdfviewer: 2.7.0
  - files_retention: 1.15.0
  - files_rightclick: 1.5.0
  - files_sharing: 1.18.0
  - files_trashbin: 1.16.0
  - files_versions: 1.19.1
  - firstrunwizard: 2.15.0
  - logreader: 2.11.0
  - lookup_server_connector: 1.14.0
  - nextcloud_announcements: 1.15.0
  - notifications: 2.14.0
  - oauth2: 1.14.0
  - password_policy: 1.16.0
  - photos: 2.2.0
  - privacy: 1.10.0
  - provisioning_api: 1.16.0
  - recommendations: 1.5.0
  - related_resources: 1.1.0-alpha1
  - serverinfo: 1.16.0
  - settings: 1.8.0
  - sharebymail: 1.16.0
  - snappymail: 2.28.1
  - support: 1.9.0
  - survey_client: 1.14.0
  - systemtags: 1.16.0
  - text: 3.7.2
  - theming: 2.1.1
  - twofactor_backupcodes: 1.15.0
  - updatenotification: 1.16.0
  - user_ldap: 1.16.0
  - user_status: 1.6.0
  - viewer: 1.10.0
  - weather_status: 1.6.0
  - workflowengine: 2.8.0
Disabled:
  - bruteforcesettings: 2.6.0
  - circles: 26.0.0 (installed 26.0.0)
  - encryption: 2.14.0
  - suspicious_login: 4.4.0
  - twofactor_totp: 8.0.0

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

{"reqId":"ZIlFLoMmYtUHgHCLRcxebgAAzQM","level":0,"time":"2023-06-14T10:12:22+05:30","remoteAddr":"10.20.4.25","user":"ecadmin","app":"webdav","method":"PUT","url":"/remote.php/dav/uploads/ecadmin/web-file-upload-0017df3152a12b2989c7740d0c8fe876-1686717741220/1","message":"Missing metadata for chunked upload","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36","version":"26.0.2.1","exception":{"Exception":"Sabre\\DAV\\Exception\\PreconditionFailed","Message":"Missing metadata for chunked upload","Code":0,"Trace":[{"file":"/web/service/apps/dav/lib/Upload/ChunkingV2Plugin.php","line":148,"function":"checkPrerequisites","class":"OCA\\DAV\\Upload\\ChunkingV2Plugin","type":"->"},{"file":"/web/service/3rdparty/sabre/event/lib/WildcardEmitterTrait.php","line":89,"function":"beforePut","class":"OCA\\DAV\\Upload\\ChunkingV2Plugin","type":"->"},{"file":"/web/service/3rdparty/sabre/dav/lib/DAV/Server.php","line":456,"function":"emit","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/web/service/3rdparty/sabre/dav/lib/DAV/Server.php","line":253,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/web/service/3rdparty/sabre/dav/lib/DAV/Server.php","line":321,"function":"start","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/web/service/apps/dav/lib/Server.php","line":366,"function":"exec","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/web/service/apps/dav/appinfo/v2/remote.php","line":35,"function":"exec","class":"OCA\\DAV\\Server","type":"->"},{"file":"/web/service/remote.php","line":172,"args":["/web/service/apps/dav/appinfo/v2/remote.php"],"function":"require_once"}],"File":"/web/service/apps/dav/lib/Upload/ChunkingV2Plugin.php","Line":284,"message":"Missing metadata for chunked upload","exception":{"header":null},"CustomMessage":"Missing metadata for chunked upload"}}
{"reqId":"ZIlFLoMmYtUHgHCLRcxebwAAzQM","level":0,"time":"2023-06-14T10:12:22+05:30","remoteAddr":"10.20.4.25","user":"ecadmin","app":"webdav","method":"PROPFIND","url":"/remote.php/dav/files/ecadmin/Test/AM-eval-6.5.1.zip","message":"File with name /Test/AM-eval-6.5.1.zip could not be located","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36","version":"26.0.2.1","exception":{"Exception":"Sabre\\DAV\\Exception\\NotFound","Message":"File with name /Test/AM-eval-6.5.1.zip could not be located","Code":0,"Trace":[{"file":"/web/service/3rdparty/sabre/dav/lib/DAV/Tree.php","line":78,"function":"getChild","class":"OCA\\DAV\\Connector\\Sabre\\Directory","type":"->"},{"file":"/web/service/3rdparty/sabre/dav/lib/DAV/Server.php","line":971,"function":"getNodeForPath","class":"Sabre\\DAV\\Tree","type":"->"},{"file":"/web/service/3rdparty/sabre/dav/lib/DAV/Server.php","line":1662,"function":"getPropertiesIteratorForPath","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/web/service/3rdparty/sabre/dav/lib/DAV/Server.php","line":1647,"function":"writeMultiStatus","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/web/service/3rdparty/sabre/dav/lib/DAV/CorePlugin.php","line":346,"function":"generateMultiStatus","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/web/service/3rdparty/sabre/event/lib/WildcardEmitterTrait.php","line":89,"function":"httpPropFind","class":"Sabre\\DAV\\CorePlugin","type":"->"},{"file":"/web/service/3rdparty/sabre/dav/lib/DAV/Server.php","line":472,"function":"emit","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/web/service/3rdparty/sabre/dav/lib/DAV/Server.php","line":253,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/web/service/3rdparty/sabre/dav/lib/DAV/Server.php","line":321,"function":"start","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/web/service/apps/dav/lib/Server.php","line":366,"function":"exec","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/web/service/apps/dav/appinfo/v2/remote.php","line":35,"function":"exec","class":"OCA\\DAV\\Server","type":"->"},{"file":"/web/service/remote.php","line":172,"args":["/web/service/apps/dav/appinfo/v2/remote.php"],"function":"require_once"}],"File":"/web/service/apps/dav/lib/Connector/Sabre/Directory.php","Line":227,"message":"File with name /Test/AM-eval-6.5.1.zip could not be located","exception":{},"CustomMessage":"File with name /Test/AM-eval-6.5.1.zip could not be located"}}

Additional info

No response

[nadi-dx]

You can try for change default max_chunk_size :

sudo -u www-data php occ config:app:set files max_chunk_size --value 2097152000

good luck

[solracsf]

Please update to 26.0.3 and report back please 👍

[sjash]

@nadi-dx : increasing the max chunks size resolved the issue.

[sjash]

Please update to 26.0.3 and report back please 👍

I upgraded the version to 26.0.3, the issue persisit. But if we manually change the chunk size suggested by @nadi-dx , it resolves the issue.

[asheroto]

Changing the chunk size did not fix for me.

~~Downgraded client to 3.9.4 and the issue went away.~~ ~~Will mention in #4826.~~

Opened separate issue https://github.com/nextcloud/desktop/issues/6079

[TheRazvy]

Nextcloud Hub 6 (27.1.1) still persist

[bjoern86]

I can confirm, the problem still exists in NC 27.1.1. As soon as the file to be uploaded exceeds the value in max_chunk_size, the upload aborts without error message. Workaround: php occ config:app:set files max_chunk_size --value 0 (Set --value 0 for no chunking at all.)

[wmeneses]

I confirm that the same thing happens to me, but changing the max_chunk_size parameters, the problem persists, I cannot upload 10MB files with the client.

Edit: I have been uploading files (100TB) from rclone and no problem, looks like it was a nextcloud client error.

Edit2:

I have gone back to version 3.9.4 and I have no problem with uploading files larger than 10MB, now what do I do with my users that are more than 600? :D

[bartcallens]

I've got the same problem on mac with client 3.10.0 . No problems on linux client 3.9.3 - fedora for the same files.

[asonix]

This happens to me on 3.10.0, but downgrading to 3.8.2 works

[wmeneses]

Hi, can I do something in the meantime, I love nextcloud, but this kind of bugs make me look bad to my users. Any manual or method I can use just to solve this problem? Sorry for insisting, thank you very much for such a wonderful product.

[asheroto]

If related to this... https://github.com/nextcloud/desktop/issues/6079 then it's a client issue so downgrading is the only solution for now. 3.9.4 works for me.

[wmeneses]

Will I have to compile this with the patch resolved? I'm still having problems with this bug, thanks a lot guys.

[okkervil]

nextcloud ios still has this issue.

[gangrif]

I have this issue, but it also happens when I upload using the web client. I don't know if this discussion is around a fix in the desktop clients, or also the web client? Can someone verify? Or is my issue different?

[asheroto]

I think some of us are describing an issue that is caused by the desktop client, and some of us are describing an issue caused by the server. The confusing part is that it's the same error message but actually two separate issues.

Although some of us initially suspected a server-side problem, the same error message actually points to a client-related issue for us. If you are having issues on the web client, then it's not a desktop client issue, but a server issue.

Steps to determine the cause: 1.) If the error happens on the web client at all, then it's a server issue, not a desktop client issue. If you're having web client issues then this thread is the best spot to discuss it. 2.) If the occurs on the desktop client but not on the web client, the issue was fixed here, just try that version. Any issues with this should probably be discussed on that issue. 3.) If the error happens on the web client and desktop client, it's still likely a server issue (because the desktop client is uploading to the server, not vice versa per the error), so the issue could exist either spot and this thread is the best spot to discuss it.

[gangrif]

I think some of us are describing an issue that is caused by the desktop client, and some of us are describing an issue caused by the server. The confusing part is that it's the same error message but actually two separate issues.

Although some of us initially suspected a server-side problem, the same error message actually points to a client-related issue for us. If you are having issues on the web client, then it's not a desktop client issue, but a server issue.

Steps to determine the cause: 1.) If the error happens on the web client at all, then it's a server issue, not a desktop client issue. If you're having web client issues then this thread is the best spot to discuss it. 2.) If the occurs on the desktop client but not on the web client, the issue was fixed here, just try that version. Any issues with this should probably be discussed on that issue. 3.) If the error happens on the web client and desktop client, it's still likely a server issue (because the desktop client is uploading to the server, not vice versa per the error), so the issue could exist either spot and this thread is the best spot to discuss it.

This is definitely best described as 3. I first noticed it after moving my nextcloud instance (which runs in containers) to a new host. When I did so I updated it. It wasnt woefully out of date or anything. I don't recall what versions i went to or from. Currently on 27.1.2.

I noticed the problem after attempting to sync a folder of photos and videos from my kids soccer game using the desktop client (windows 11, i believe the client is 3.10, but cant seem to figure out where to find the version in the client), none of the videos would upload, but the pictures did fine. Some trial and error, i found that around 10mb was the point where uploads started to fail, and i also couldn't upload via the web ui.

The client and the server both give me very similar errors. The one about the metadata at the top, and then a not found error on whatever file i was uploading.

The web client, while uploading, just seems to silently fail. I get a progress bar for the upload, then it just goes away, and the file doesnt upload. I have to set server log level 0 to get any error at all out of it in the logs.

Let me know if I can provide any more info.

Thanks!

[cfiehe]

We are facing the same issue that is reported here. I hope that we have found the root cause 😃. I have created a pull request (https://github.com/nextcloud/server/pull/41149). We have tested the fix in our productive environment for some days and it works properly in combination with our local Cloudian S3 object storage. I would be nice, if someone could acknowledge that this fix also works in his environment when large files are being uploaded. Thanks a lot.

[solracsf]

I don't believe your patch "fixes" anything; it will trow a NotFound and fallback to v1. This why it "works".

[cfiehe]

It's a pity 😢.

@solracsf Have you got an idea what problem we are facing here?

[cfiehe]

@solracsf Yes, you are right. I missed that NotFound-exception. It was a typo in the Nextcloud configuration file. For details please have a look at: https://github.com/nextcloud/server/pull/41149#discussion_r1374552625.

[joshtrichards]

@sjash Does the problem go away if you configure a memcache? You appear to lack any memcache.* parameters in your config.

https://docs.nextcloud.com/server/latest/admin_manual/configuration_server/caching_configuration.html#recommendations-based-on-type-of-deployment

[sjash]

@sjash Does the problem go away if you configure a memcache? You appear to lack any memcache.* parameters in your config.

https://docs.nextcloud.com/server/latest/admin_manual/configuration_server/caching_configuration.html#recommendations-based-on-type-of-deployment

@joshtrichards Adding Memcache has earlier resulted in file locking in my environments. Additionally, I do not see a direct link where part of file which where not yet uploaded/ to-be uploaded in NC to have a relation with memcache.

I recommend for thorough checking of client version 3.10.X because after setting the chunk size as suggested by @nadi-dx , I'm able to upload file. But when I try the same using client version >= 3.10.X it results the same error to client.

[joshtrichards]

The metadata is stored in the cache.

The chunk size change works around the issue, but doesn't solve it. You'll just run into it at a different file size and/or be limited in what file sizes you can upload (by your web server/proxy configuration). The reason you're seeing it only in the newer client version(s) is because that's when support for v2 chunked upload streaming for s3 was added (nextcloud/desktop#5819).

#41198 is a follow-up (not yet implemented) that will likely tackle this in environments where caching isn't functioning by falling back on the old behavior. But that also means lower performance and higher disk usage (lack of #27034).

I'm not sure what the locking matter is you're referring to, but you can just disable locking if need be.

[maius]

I think some of us are describing an issue that is caused by the desktop client, and some of us are describing an issue caused by the server. The confusing part is that it's the same error message but actually two separate issues. Although some of us initially suspected a server-side problem, the same error message actually points to a client-related issue for us. If you are having issues on the web client, then it's not a desktop client issue, but a server issue. Steps to determine the cause: 1.) If the error happens on the web client at all, then it's a server issue, not a desktop client issue. If you're having web client issues then this thread is the best spot to discuss it. 2.) If the occurs on the desktop client but not on the web client, the issue was fixed here, just try that version. Any issues with this should probably be discussed on that issue. 3.) If the error happens on the web client and desktop client, it's still likely a server issue (because the desktop client is uploading to the server, not vice versa per the error), so the issue could exist either spot and this thread is the best spot to discuss it.

This is definitely best described as 3. I first noticed it after moving my nextcloud instance (which runs in containers) to a new host. When I did so I updated it. It wasnt woefully out of date or anything. I don't recall what versions i went to or from. Currently on 27.1.2.

I noticed the problem after attempting to sync a folder of photos and videos from my kids soccer game using the desktop client (windows 11, i believe the client is 3.10, but cant seem to figure out where to find the version in the client), none of the videos would upload, but the pictures did fine. Some trial and error, i found that around 10mb was the point where uploads started to fail, and i also couldn't upload via the web ui.

The client and the server both give me very similar errors. The one about the metadata at the top, and then a not found error on whatever file i was uploading.

The web client, while uploading, just seems to silently fail. I get a progress bar for the upload, then it just goes away, and the file doesnt upload. I have to set server log level 0 to get any error at all out of it in the logs.

Let me know if I can provide any more info.

Thanks!

I would like to add 4. - Mobile client. Setup:

• Nextcloud Subscription at German Hoster
• Version: 26.0.4 Enterprise (Client: NC Liquid for iOS 4.9.1.0)
• IPhone SE 2020

Just ran into the following problem:

• Attempt to upload Video (166 MB) from Iphone to NC
• Error Message "Missing metadata for chunked upload -9993 (keeps popping up every 5 sec)
• Cannot abort upload (Upload not shown in "transfer" or "activity" section of the client)
• Upload can only be aborted by deleting the cache of the client

I can provide screenshots, client protocol and server protocol if needed.

[wintix]

The metadata is stored in the cache.

The chunk size change works around the issue, but doesn't solve it. You'll just run into it at a different file size and/or be limited in what file sizes you can upload (by your web server/proxy configuration). The reason you're seeing it only in the newer client version(s) is because that's when support for v2 chunked upload streaming for s3 was added (nextcloud/desktop#5819).

We just had the same issue, fresh install, activating Redis solved the s3 chunking issue immediately.

[ZerhackerJagi]

We just had the same issue, fresh install, activating Redis solved the s3 chunking issue immediately.

Does this mean you got it working using Redis? I'm also running Redis nearly plain out of the box (just installed it and configured nextcloud to use it) and still got the "missing metadata for chunked upload" error. Had a fresh install of nextcloud (27.1.3) yesterday. I'm receiving the error by using the iOS App of nextcloud (Version 4.9.1.0)

[wintix]

Yes, you need a memory Cache for the Chunks not to get lost.

[ZerhackerJagi]

did you make changes to the /etc/redis/redis.conf ?

[gangrif]

can confirm, simply enabling redis did nothing for my issue. I: added a redis container to my pod pointed nextcloud at it following the basic redis setup documented here: https://docs.nextcloud.com/server/19/admin_manual/configuration_server/config_sample_php_parameters.html

Issue is exactly the same as before. Is more than the basic configuration required here?