Bulk upload issues

[joshtrichards]

(WIP: will be updated as needed)

Summary: Indications are that there are one or more scenarios that trigger problems with bulk uploads, creating serious performance problems (e.g. 100% CPU usage, non-responsive). Many find success through workarounds like disabling bulk uploads or adjusting upload limits. Some reporters have noted that #5680 resolved their situation, while others it has not. There may either be additional triggers or simply multiple unrelated root causes.

Note: Many of these are likely duplicates (which is part of what needs to get sorted out)

Issues:

• #5726
  • OP reports fixed (by #5680): https://github.com/nextcloud/desktop/issues/5726#issuecomment-1722179739)
    • Should likely be closed as a result; new reporters clearly have different circumstances so should create new Issue(s) as needed
• #5094
  • OP reports fixed (by #5680): https://github.com/nextcloud/desktop/issues/5094#issuecomment-1637072018
    • OP reports still fixed: https://github.com/nextcloud/desktop/issues/5094#issuecomment-2249953808
    • Should stay closed as a result; new reporters clearly have different circumstances so should create new Issue(s) as needed
• #5031
• #5349
• #4106
  • OP reports fixed (by #5680): https://github.com/nextcloud/desktop/issues/4106#issuecomment-1620373666
• #5888
• #6511
• nextcloud/server#29702
• #6889
• #6941

PRs:

• #5680
• https://github.com/nextcloud/desktop/pulls?q=bulk+is%3Apr+is%3Aclosed+

[sapphirepro]

Yeah, when uploading bulk of files, it wastes cpu and after a tiny upload goes rescan etc in a loop. With fiber network uploading like 1 GB of photos can take even an hour to complete. So solution still needed

[mgallien]

thanks for the help raising the importance of this is there known reproducible scenario that are still failing with recent server and client releases ? no big hopes of a quick fix if I first need to investigate and spend hours trying to analyze possible ways to trigger the bugs

[sapphirepro]

thanks for the help raising the importance of this is there known reproducible scenario that are still failing with recent server and client releases ? no big hopes of a quick fix if I first need to investigate and spend hours trying to analyze possible ways to trigger the bugs

Ok. Giving information. Latest NextCloud stable release. Latest Linux desktop client. Now situation is that to client side dropped 700 mb of files (approx 1-1.5 Mb per file, photos). So key scenario to test reasoanble size like 700 MB or above, while they should be small and a lot of them, like 100 files or above. Sync will be like dropping fast maybe 30-40 mb, then huuuge wait like few minutes, then again short time upload of 30-40 mb and so on in the loop till all uploaded.

Expected behavious is if dropped 700mb of many files to upload, then upload them all non stop and only then resync. What I suspect as a reason of this performance bug, is that it starts upload, uploads tiny files like 40 mb in total and then server reports changes (those same files), upload gets held, resync and then proceed. It should throw all files in a bunch and only then resync something. Also a tray icon and clicking on it while such sync doesn't often show floating window of nextcloud desktop client, by which I assume that there is process violation as well, that sync or something being done in main process and blocking UI, while it should be a separate thread.

I tried the best I could to explain. If any more info needed, do tell, I will try to assist.

[Bockeman]

My Windows10 Nextcloud client is 3.13.2 (Stable). I ugraded my (Fedora 40)server to Nextcloud Hub 8 (30.0.0 RC1) (Beta and soon to be RC2). Consequently, I got loads of errors and warnings which I have largely eliminated apart from "Error no app in context Computed md5 hash is incorrect" which appears to be related to this topic.

I have "'bulkupload.enabled' => false," in my server nextcloud.cfg (though I am not convinced this is effective).

My Windows10 Nextcloud client settings shows 20TB in use, and this includes Sync Connections for AppData/Roaming. The important point being that files are changing constantly at the whim of Windows.

Bulk transfer seems a good option IMHO, but there is a high likelyhood of file changes before the transfer is complete. Hence I am expecting all files uploaded to the server to retain the file modified date, as captured in the bulk, and a subsequent transfer to handle files changed since the previous bulk capture.

I am getting "Error no app in context Computed md5 hash is incorrect" every 10 minutes or so, therefore I think I have a suitable environment for testing possible fixes/patches.

Please advise if there is any patch you wish me to apply to verify possible solutions.

[keithkhl]

another frustrated user here. thought it is due to misconfiguration and wasted days, if not weeks.

One more catch for bulk uploading files. When files are inside the nested folders, it does not copy any files. It only copies folder structure.

it is not even browser or OS issue. For the last a month or so, I've tried bare metal installation to AIO, and even on windows.

OneDrive used to be notoriously slow, but it got way faster now. Yet, many people still stick to Dropbox's paid option because of the performance. Gotta tell you that what I see so far for the last a month is beyond expectation. It's not like my server is not equipped. You don't usually assign a 64-core + 256GB RAM w/ 10TB NVMe and 10Gbps just for an ERP type service. Give us at least a reasonable option to expand bandwidth and max use of server resources.

[sol8712]

#7615 Another one to add to the random bulk uploads issues.

Edit: Disabling bulk uploads does not effect my issue of NC Desktop filling RAM until PC crash.

[timinator2020]

Having the same issue - added about 3000 files, but all totaling only 32MB. It uploads 100 files, then pauses for 30-60 seconds (no network activity). After 30-60 seconds, it instantly copies another 100 files and pauses again.

I have tried changing the minChunkSize/maxChunkSize values in the desktop config.ini (even went so far as to set the minChunkSize value to 1). I disabled the redis container to rule that out (this is mostly just for me, so file locking isn't a big concern). I added some timeout settings to the fastcgi nginx proxy. I also tried a few other things, but I can't remember what they were at the moment. They didn't help either.

The only thing that seems to help is adding 'bulkupload.enabled' => false to config.php (which completely disables the bulk-upload functionality). Transfer rates are slower, but at least they don't freeze (so the total transfer time is less). I'd really like to see the issue fixed properly, since it would definitely help with use cases like this (thousands of tiny files).

[tlemarchand]

Same issue on nextcloud-client 3.14.2, with a 7GB folder of photos. After hours being stucks, and multiple retries, I tried 'bulkupload.enabled' => false on my server, restarted sync, then it finished sync in a few minutes.

[Thomas131]

Same here. A few hundred new files (~10 GB). I killed nextcloud and restarted. After a few seconds everything froze, consuming ~75-80% CPU.

Nextcloud version 3.14.4
Git revision 1d624144b4170893eb119d8e09abb9f4a60d6f2a
Using Qt 6.8.0, built against Qt 6.8.0
Using Qt platform plugin 'xcb'
Using 'OpenSSL 3.3.2 3 Sep 2024'
Running on NixOS 24.11 (Vicuna), x86_64

The last loglines before the freeze (I anonymized the information and removed duplicates):

2025-01-06 22:01:59:951 [ info nextcloud.gui.folderwatcher /build/source/src/gui/folderwatcher.cpp:252 ]:	Detected changes in paths: QSet([416 files])
2025-01-06 22:01:59:972 [ info nextcloud.gui.folder /build/source/src/gui/folder.cpp:641 ]:	Ignoring spurious notification for file "path/to/a/file.m4b"
[Same message for 270 files]
2025-01-06 22:02:00:057 [ debug nextcloud.sync.database /build/source/src/common/syncjournaldb.cpp:3062 ]	[ OCC::SyncJournalDb::commitInternal ]:	Transaction commit "Upload info" and starting new transaction
2025-01-06 22:02:00:057 [ info nextcloud.sync.propagator.bulkupload /build/source/src/libsync/bulkpropagatorjob.cpp:189 ]:	"/some/path/file48.mp3" transmission checksum "[md5 hash I guess]" "/home/username/nc/some/path/file48.mp3"
[Same 2 lines for 79 files]

And I just noticed (at least once) this error appearing in the console completely at nextcloud start (before complaining about config-folder migration):

Failed to initialize instances shared memory:  "QSharedMemory::handle: doesn't exist"

'bulkupload.enabled' => false on the server fixed it.

Best Regards, Thomas

[muonicpress]

How frustrating that this still exists years later. I'm moving from NC to NC AIO on a new server (latest client and AIO versions) and only remembered just now (after 2 days of frustration) that this was an issue in the past. I disabled bulk upload on the server and now my client is speeding through the files.

[Xoc82]

I encountered the same issue, I had to disable bulk upload, this bug should get a high priority! Bug like this can make nextcloud unusable.

[Zero-33]

I'll add +1 - same issue, same work around.

It sounds like a tricky issue to resolve, its just in a production environment it is undermining confidence that all of your files are indeed synced on the server, because for me the desktop app shows green ticks across the board. So naturally you think everything is fine.

When working with others sharing/syncing files, confidence in those green ticks erodes quickly with this issue.

To help any who find this, save you some time researching and (as I am) are new to docker, here's a quick how to for this work around:

The only thing that seems to help at this time is adding 'bulkupload.enabled' => false to config.php

How to:

• go into the bash of the nextcloud container

docker exec -it nextcloud-aio-nextcloud bash

• edit config.php
  • Location for me was here - /var/www/html/config
  • open file in vi

vi /var/www/html/config/config.php

• the file should looks approx like this (this was the beginning of mine)

<?php 
$CONFIG = array ( 
     'one-click-instance' => true,
     'one-click-instance.user-limit' => 100,
    'memcache.local' => '\\OC\\Memcache\\APCu',
   
   // lots of other existing configurations 
	
);

• add this line in this section - 'bulkupload.enabled' => false
• I added near the top so I could see it

<?php                                                                
$CONFIG = array (                                                    
  'one-click-instance' => true,                              
  'one-click-instance.user-limit' => 100,                        
  'memcache.local' => '\\OC\\Memcache\\APCu',     
  'bulkupload.enabled' => false,                     
  'apps_paths' =>                                                
  array ( 
 // lots of other configs
);

• :wq to save the file and exit vi
• exit docker bash, by simply typing exit
• I then stopped all containers and started them up again, via AIO Web Interface (you probably only need to restart the nextcloud container)

Hope that helps make it easier to get this work around for those new to this.

If I can help test any possible future fixes to this issue, let me know

[restuccia]

Hi, this is a major issue for me as well, especially when migrating all my files to Nextcloud. And sadly, running Nextcloud on TrueNAS prevents me from manually editing the config.php. Do you know whether I can disable bulkupload some other way via TrueNAS?

[SwarzKalman]

I'll add +1 - same issue, same work around.

It sounds like a tricky issue to resolve, its just in a production environment it is undermining confidence that all of your files are indeed synced on the server, because for me the desktop app shows green ticks across the board. So naturally you think everything is fine.

When working with others sharing/syncing files, confidence in those green ticks erodes quickly with this issue.

To help any who find this, save you some time researching and (as I am) are new to docker, here's a quick how to for this work around:

The only thing that seems to help at this time is adding 'bulkupload.enabled' => false to config.php

How to:

• go into the bash of the nextcloud container

docker exec -it nextcloud-aio-nextcloud bash

• edit config.php

  • Location for me was here - /var/www/html/config
  • open file in vi

vi /var/www/html/config/config.php

• the file should looks approx like this (this was the beginning of mine)

true, 'one-click-instance.user-limit' => 100, 'memcache.local' => '\\OC\\Memcache\\APCu', // lots of other existing configurations ); * add this line in this section - `'bulkupload.enabled' => false` * I added near the top so I could see it true, 'one-click-instance.user-limit' => 100, 'memcache.local' => '\\OC\\Memcache\\APCu', 'bulkupload.enabled' => false, 'apps_paths' => array ( // lots of other configs ); * :wq to save the file and exit vi * exit docker bash, by simply typing `exit` * I then stopped all containers and started them up again, via AIO Web Interface (you probably only need to restart the nextcloud container) Hope that helps make it easier to get this work around for those new to this. If I can help test any possible future fixes to this issue, let me know

thank you, it helped me on my windows client (3.14.3 and 3.15.3 versions) that was my config file location at my OMV based Portrainer container:

/config/www/nextcloud/config# nano config.php

[talesam]

I'm facing the same issue, and it seems like it's been a long time without a solution. It pushes my CPU to 100%, and look at my CPU temperature—it goes over 95 degrees. It's frustrating that something so critical has been left unresolved for so long.

[Green2Matter]

Similar is on MacOS, I use AIO; all latest and greatest. When running in cli to check log:

~ % /Applications/nextcloud.app/Contents/MacOS/nextcloud --logfile nextcloud.log
nextcloud.gui.application: Migrating old config from "/Users/mar/Library/Application Support/Nextcloud" to "/Users/mar/Library/Preferences/Nextcloud"
nextcloud.gui.application: Failed to move the old config directory to its new location ( "/Users/mar/Library/Application Support/Nextcloud" to "/Users/mar/Library/Preferences/Nextcloud" )
nextcloud.gui.application: Will move the individual files QList("Documents_sync.log", "Drive_sync.log")
nextcloud.gui.application: Fallback move of  "Documents_sync.log" also failed
nextcloud.gui.application: Fallback move of  "Drive_sync.log" also failed
2025-03-10 21:48:45.260 nextcloud[5331:377520] +[IMKClient subclass]: chose IMKClient_Modern
2025-03-10 21:48:45.260 nextcloud[5331:377520] +[IMKInputSession subclass]: chose IMKInputSession_Modern

At first I thought it is related to "failed migration" but it wasn't. After adding: 'bulkupload.enabled' => false, to config.php nextcloud.app finally stopped freezing on startup (beginning of sync process).

[MaxLux]

I'll add +1 - same issue, same work around.

It sounds like a tricky issue to resolve, its just in a production environment it is undermining confidence that all of your files are indeed synced on the server, because for me the desktop app shows green ticks across the board. So naturally you think everything is fine.

When working with others sharing/syncing files, confidence in those green ticks erodes quickly with this issue.

To help any who find this, save you some time researching and (as I am) are new to docker, here's a quick how to for this work around:

The only thing that seems to help at this time is adding 'bulkupload.enabled' => false to config.php

How to:

* go into the bash of the nextcloud container

docker exec -it nextcloud-aio-nextcloud bash

* edit config.php
  
  * Location for me was here - `/var/www/html/config`
  * open file in vi

vi /var/www/html/config/config.php

* the file should looks approx like this (this was the beginning of mine)

true, 'one-click-instance.user-limit' => 100, 'memcache.local' => '\\OC\\Memcache\\APCu', // lots of other existing configurations ); * add this line in this section - `'bulkupload.enabled' => false` * I added near the top so I could see it true, 'one-click-instance.user-limit' => 100, 'memcache.local' => '\\OC\\Memcache\\APCu', 'bulkupload.enabled' => false, 'apps_paths' => array ( // lots of other configs ); * :wq to save the file and exit vi * exit docker bash, by simply typing `exit` * I then stopped all containers and started them up again, via AIO Web Interface (you probably only need to restart the nextcloud container) Hope that helps make it easier to get this work around for those new to this. If I can help test any possible future fixes to this issue, let me know

Thank You!! Helped me with the issue on Nextcloud 31.0.2 (AIO) and Windows Client 3.16.2

[Rello]

With the upcoming release of the desktop client (version 3.16.3), the bulk upload feature will be temporarily disabled.

We understand that some of you may rely on this feature, and this decision wasn't made lightly. However, we've encountered ongoing technical issues involving the interaction between the client, server, and underlying systems—including network stack and Qt dependencies—that impact the stability of the feature.

In prioritizing a smoother and more reliable experience for all users, we believe it's best to disable bulk upload for now. The overall benefit of the feature does not currently outweigh the potential for disruption.

If you're unable to update to version 3.16.3 right away, we recommend manually disabling the feature on the server side to avoid any issues: 🔧 How to disable bulk upload on the server →

We appreciate your understanding and are continuing to work on improving the experience.

[xrogeman]

I added “bulkupload.enabled”: false, to my config.php and didnt help with the crash, I have a large folder of 5000 files and if I try to paste ir to the nextcloud folder using the windows client, the client app crashes inmediately if I remove the folder and open the windows client again It syncs without a problem

this is the begining of the config i use

occ $ config:list system { “system”: { “htaccess.RewriteBase”: “/”, “memcache.local”: “\OC\Memcache\APCu”, “bulkupload.enabled”: false, “apps_paths”: [ … …

in currently running windows 11, Nextcloud Desktop Client Version 3.16.4 (Windows) Nextcloud 31.0.5 server in a docker container

checking the log found at C:\Users\user1\AppData\Roaming\Nextcloud\logs\20250521_1608_nextcloud.log.2 seems like is trying to sync files but crashes in the process, using the web browser those files never appear these are the last lines in the log:

“C:/Users/user1/Nextcloud/docs/EM/g/20220709 CT scanner (VersionWEB)/1.2.840.113619.6.95.31.0.3.4.1.9172.13.1697043/CTL600783_P L TC ABD PELVIS MULTIC SIMPL/1_205.dcm”, “C:/Users/user1/Nextcloud/docs/EM/g/20220709 CT scanner (VersionWEB)/1.2.840.113619.6.95.31.0.3.4.1.9172.13.1697043/CTL600783_P L TC ABD PELVIS MULTIC SIMPL/4_50.dcm”, “C:/Users/user1/Nextcloud/docs/ID/Ro/i/ia.jpg”, “C:/Users/user1/Nextcloud/docs/EM/g/20220709 CT scanner (VersionWEB)/1.2.840.113619.6.95.31.0.3.4.1.9172.13.1697043/CTL600783_P L TC ABD PELVIS MULTIC SIMPL/1_318.dcm”, “C:/Users/user1/Nextcloud/docs/EM/g/20220502 1.3.76.13.1060.2.20220502093926.100462293.1/05 Video.mp4”)

my php config:

root@60355b1a136e:/usr/local/etc/php/conf.d# cat nextcloud.ini memory_limit=2G upload_max_filesize=16G post_max_size=16G max_input_time=3600 max_execution_time=3600

any idea how to fix it?

[TheThief]

@Rello with this bug being closed, does that mean there's no intention to fix and re-enable bulk upload in the future?

[Rello]

Hi, It will be picked up again. But it has no priority at the moment and there is no added calie leaving the tickets open. It will require rework in client and server.

Once technically available, we will enable the feature again

Thank you