nextcloud
[Bug]: Linux Desktop Client is constantly syncing and writing to disc even no changes happened
⚠️ Before submitting, please verify the following: ⚠️
- [x] This is a bug, not a question or a configuration issue.
- [x] This issue is not already reported on Github (I've searched it).
- [x] Nextcloud Server and Desktop Client are up to date. See Server Maintenance and Release Schedule and Desktop Releases for supported versions.
- [x] I agree to follow Nextcloud's Code of Conduct
Bug description
Since one of the last two updates (3.14.1 or 3.14.0) the nextcloud desktop client is syncing every second. This means that the icon changes from green with white checkmark to the blue syncing sign for about half a second, changes back to green sign and again after half of a second back to the syncing sign.
In the system monitor, the nextcloud desktop app writes constantly 1.7 mb/s to disk, which is about 50 GB on a full working day. This happens all the time, even no program is open or writing to the nextcloud folder. It does not occure, when no network connection is present. So maybe it could also be a problem on the nextcloud server.
Steps to reproduce
- Check that a network connection to the server is possible
- Start the nextcloud desktop app
Expected behavior
The nextcloud desktop app should only sync, when a new file is created or a file has changed.
Which files are affected by this bug
Don't know which files are affected
Operating system
Linux
Which version of the operating system you are running.
ZorinOS 17.2
Package
Community FlatPak
Nextcloud Server version
29.0.8
Nextcloud Desktop Client version
3.14.1
Is this bug present after an update or on a fresh install?
Updated from a minor version (ex. 3.4.2 to 3.4.4)
Are you using the Nextcloud Server Encryption module?
Encryption is Disabled
Are you using an external user-backend?
- [x] Default internal user-backend
- [ ] LDAP/ Active Directory
- [ ] SSO - SAML
- [ ] Other
Nextcloud Server logs
No response
Additional info
While this bug is present, I have updated the nextcloud server version from 29.0.7 (i guess) to 29.0.8 but the bug still exists. I am not a 100% sure if I have updated the desktop client as well, but I think the problem started with version 3.14.0 and I have updated it to 3.14.1 without success. I think the problem started after a desktop client update and not after a server update.
I have also tried to reinstall the nextcloud desktop client and deleted the local copies of all files and resynced them without success as well.
I am not syncing local data of the nextcloud server, the data are connected by smb to the nextcloud server from a Qnap NAS. I connected it via the GUI of Nextcloud and not via fstab or something on the linux server where nextcloud is installed.
Same or similar issue here. For me its syncing every ~7 seconds. I'm using v 3.14.1 Regarding the huge writes to disk, please look at #5302. I managed to disable the logging, but it still sync every ~7 seconds.
#=#=#=# Syncrun started 2024-10-16T12:22:25Z
#=#=#=#=# Propagation starts 2024-10-16T12:22:28Z (last step: 2802 msec, total: 2802 msec)
#=#=#=# Syncrun finished 2024-10-16T12:22:29Z (last step: 1393 msec, total: 4196 msec)
#=#=#=# Syncrun started 2024-10-16T12:22:32Z
#=#=#=#=# Propagation starts 2024-10-16T12:22:35Z (last step: 2786 msec, total: 2786 msec)
#=#=#=# Syncrun finished 2024-10-16T12:22:37Z (last step: 1371 msec, total: 4158 msec)
#=#=#=# Syncrun started 2024-10-16T12:22:40Z
#=#=#=#=# Propagation starts 2024-10-16T12:22:43Z (last step: 2896 msec, total: 2896 msec)
#=#=#=# Syncrun finished 2024-10-16T12:22:44Z (last step: 1521 msec, total: 4418 msec)
#=#=#=# Syncrun started 2024-10-16T12:22:48Z
#=#=#=#=# Propagation starts 2024-10-16T12:22:50Z (last step: 2732 msec, total: 2732 msec)
#=#=#=# Syncrun finished 2024-10-16T12:22:52Z (last step: 1332 msec, total: 4064 msec)
#=#=#=# Syncrun started 2024-10-16T12:22:55Z
#=#=#=#=# Propagation starts 2024-10-16T12:22:58Z (last step: 2794 msec, total: 2794 msec)
#=#=#=# Syncrun finished 2024-10-16T12:22:59Z (last step: 1412 msec, total: 4206 msec)
#=#=#=# Syncrun started 2024-10-16T12:23:02Z
#=#=#=#=# Propagation starts 2024-10-16T12:23:05Z (last step: 2788 msec, total: 2788 msec)
#=#=#=# Syncrun finished 2024-10-16T12:23:07Z (last step: 1414 msec, total: 4203 msec)
Update: I downgraded temporarily to version 3.13.4 but its the same here.
hmmm, I am wondering if my ssd-lifespan got eaten by the log files since the beginning of using nextcloud. I am also wondering, why they ignore this since years, because this can heavily impact the lifespan of all user's ssd.
I checked it on my machine and it seems to write a lot of log-files as well. It also compresses the logfile 25 times a minute. So every minute I get 25 new .gz files. Due to other users posts and bug-reports, I think this problem could be existing since I started using nextcloud a couple of years ago.
On the other hand, the syncing problem exists only since one of the last two updates. I have never seen changing the symbol so frequently. Normally the sync process only starts when new files saved to the nextcloud folder or existing ones got updated.
Currently the nextcloud app is on "pause sync", so no log files really will be written. When I write/edit any file I manually start syncing once. This really is a pain in the ass. But the real problem is not the syncing, it's more the huge amount of files written. I thought this is just one problem, but know I think when the sync problem is solved, the writing problem could still exist.
How do you disabled the logging? Like how the user thvitt did it on the bugreport you mentioned, or another way?
See my comment here https://github.com/nextcloud/desktop/issues/5302#issuecomment-2416475159
I had some errors in the server side-protocol, notably missing permissions on directories which blocked full scanning of files. I fixed them all and now... my client no longer syncs every x seconds. So it looks like the server is triggering all these syncs.
I had some errors in the server side-protocol, notably missing permissions on directories which blocked full scanning of files. I fixed them all and now... my client no longer syncs every x seconds. So it looks like the server is triggering all these syncs.
Hi, seems to be the same use case than mine. What permissions have been missing in your Samba server? I am running Samba in docker environment.
Greets Thorsten
I tried to only sync a local nextcloud server folder, then the sync and write problem were gone. A nex logfile was written once every several minutes or so and not 25 times a minute. To be honest, there was only 1 file in it, while there are hundreds and thousands in the smb-folder I normally sync.
On the Qnap nas (smb-server) the share has read/write/execute for owner/group/others (777 in linux). The user who accesses the share has read/write (execute cannot be configured) access. There aren't really more things you can change on the GUI. So I am also really interessted in what you have changed.
I do not use Samba, I use local external storage. This storage contained some sub-folders that were not accessible to the Nextcloud user. As a result, the file scan couldn't complete. I changed the file permissions and now the file scan is complete and the constant syncing on the client side is gone.
If you already have 777 on all files and (sub-)folders, there may be another problem on your end. You should look at the server protocol and try to resolve any errors listed there.
Yesterday I realized, that after I changed the folder to synchronize from the smb-folder to the local nextcloud folder and then changing back, it does not sync the whole smb-folder again. There were some files missing, even the client doesn't show anything. Maybe I have to reproduce that and create a new bug report.
Back to this issue, I changed the log-level on the server to debug and let the client active for about 30 seconds as it produces a lot of entries and viewing the server log became really slow and laggy since they changed it a couple of versions ago. I know I can download the log and have a view in another program, but for this time it was ok. The log only showed some dirty tables read. Don't know if they have anything to to with the client.
dirty table reads: SELECT `filecache`.`fileid`, `storage`, `path`, `path_hash`, `filecache`.`parent`, `filecache`.`name`, `mimetype`, `mimepart`, `size`, `mtime`, `storage_mtime`, `encrypted`, `etag`, `filecache`.`permissions`, `checksum`, `unencrypted_size`, `metadata_etag`, `creation_time`, `upload_time`, `meta`.`json` AS `meta_json`, `meta`.`sync_token` AS `meta_sync_token` FROM `*PREFIX*filecache` `filecache` LEFT JOIN `*PREFIX*filecache_extended` `fe` ON `filecache`.`fileid` = `fe`.`fileid` LEFT JOIN `*PREFIX*files_metadata` `meta` ON `filecache`.`fileid` = `meta`.`file_id` WHERE `filecache`.`parent` = :dcValue1 ORDER BY `name` ASC
dirty table reads: SELECT `name`, `propertypath`, `propertyname`, `propertyvalue`, `valuetype` FROM `*PREFIX*filecache` `f` LEFT JOIN `*PREFIX*properties` `p` ON (`propertypath` = CONCAT(:dcValue1, `name`)) AND (`userid` = :dcValue2) WHERE `parent` = :dcValue3
dirty table reads: SELECT `s`.*, `f`.`fileid`, `f`.`path`, `f`.`permissions` as `f_permissions`, `f`.`storage`, `f`.`path_hash`, `f`.`parent` as `f_parent`, `f`.`name`, `f`.`mimetype`, `f`.`mimepart`, `f`.`size`, `f`.`mtime`, `f`.`storage_mtime`, `f`.`encrypted`, `f`.`unencrypted_size`, `f`.`etag`, `f`.`checksum`, `st`.`id` AS `storage_string_id` FROM `*PREFIX*share` `s` LEFT JOIN `*PREFIX*filecache` `f` ON `s`.`file_source` = `f`.`fileid` LEFT JOIN `*PREFIX*storages` `st` ON `f`.`storage` = `st`.`numeric_id` WHERE (`s`.`file_source` = :dcValue1) AND (`s`.`share_type` = :dcValue2) AND (`s`.`share_with` IN (:dcValue3)) AND ((`s`.`item_type` = :dcValue4) OR (`s`.`item_type` = :dcValue5)) ORDER BY `s`.`id` ASC
Any ideas?
I'm also seeing this issue with "external storage" folders. In my case, my external storage is an S3 bucket. The fact that it also impacts S3 bucket external storage, makes me think it's a broader issue.
Desktop app: 3.14.1 (Flatpak)
Server: 29.0.4 (Docker)
My server does use SQLite for the database since it's a single-user instance. This is the only file-sync issue I've had.
Hi, I use a Samba Docker container for Nextcloud as an external drive. If I now enable this external drive to be synchronization with my Liux desktop client for Netxcoud, the 2s autosync problem arises.
The Samba Docker log then shows then: idmap range not specified for domain '*'
Does anyone have an idea?
Hi,
I only have external storage linked to Nextcloud via SMB.
For now, I've temporarily downgraded to version 3.12.3 (Flatpak). It's not a permanent solution either... I can't say why this version works, but maybe it’ll help someone.
Nextloud works fine on Mac and Windows, but on Linux it's not working with the newer versions.
I also see the "re-sync every other second" issue for one of my instances, but I realized another cause: Do you by any chance use shares with read-only permission? If so, please check if the issue is resolved by these steps:
- make sure all local files are also on the server
- remove all read-only folders from the desktop settings
- stop the desktop client
- move the corresponding local folders aside (outside any synced folder)
- restart the desktop client
If you now no longer see the "sync every other second" issue, the local write permissions (and their handling by the client) are the culprit. Probably check our https://github.com/nextcloud/desktop/issues/7318. This goes back on https://github.com/nextcloud/desktop/issues/6296 which landed in 3.13 in March 2024. I just filed https://github.com/nextcloud/desktop/issues/7586 "Issue 2" describes in detail what I was referring to here, but this seems to be 3.15+ specific.
For me, the answer is "no". I am the only one using nextcloud and I configured read/write permissions in the smb-connector on the nextcloud GUI. The underlying QNAP NAS also has read/write permission for the user, who connects from nextcloud to Qnap. So I should have read/write permissions on any folder/file.
In my last post I have written, that after changing to a local nextcloud folder only, the sync worked, but after changing back to the smb folder, not all files were syncing correctly. So something under the hood seems to make problems, but I had no time yet to really check that again and create a bug report. But both problems could be related.
Back to the problem about the excessive syncing, I get several log entries for ALL files on the nextcloud. So more files you have, more log entries exist. Here are some important log-entry-examples. The first couple of lines show the start of the sync process, while line 7 has a list of all files and folders, which are several hundred. The three lines in the middle exist for all files in the nextcloud I think and the last line shows the successful sync status.
The logfile has a size of about 10 MB with thousands of lines an will be created about 25 times a minute. Hard to analyse that to find out what the problem could be. May someone else have an idea, otherwise the desktop client is nearly useless as currently I only sync once a day.
[ info nextcloud.gui.folder.manager /run/build/nextcloud-client/src/gui/folderman.cpp:702 ]: Schedule folder "1" to sync!
[ info nextcloud.gui.application /run/build/nextcloud-client/src/gui/owncloudgui.cpp:251 ]: Sync state changed for folder "$url$" : "Not yet started"
[ info nextcloud.gui.folder.manager /run/build/nextcloud-client/src/gui/folderman.cpp:883 ]: Starting the next scheduled sync in 0 seconds
[ info nextcloud.gui.application /run/build/nextcloud-client/src/gui/owncloudgui.cpp:251 ]: Sync state changed for folder "$url$" : "Preparing to sync"
[ info nextcloud.gui.folder /run/build/nextcloud-client/src/gui/folder.cpp:1046 ]: *** Start syncing "$url$" - Nextcloud client version 3.14.3daily
[ info nextcloud.gui.folder /run/build/nextcloud-client/src/gui/folder.cpp:1080 ]: Allowing local discovery to read from the database
[ info nextcloud.sync.engine /run/build/nextcloud-client/src/libsync/syncengine.cpp:1220 ]: paths to discover locally $all_files_and_folders$
[ info nextcloud.sync.discovery /run/build/nextcloud-client/src/libsync/discovery.cpp:86 ]: STARTING "$folder$" OCC::ProcessDirectoryJob::ParentNotChanged "$folder$" OCC::ProcessDirectoryJob::NormalQuery
[ info nextcloud.sync.discovery /run/build/nextcloud-client/src/libsync/discovery.cpp:550 ]: "Processing \"$file$" | (db/local/remote) | valid: true/true/db | mtime: 1674661555/1674661555/0 | size: 87804/87804/0 | etag: \"645b41adc0401\"//\"\" | checksum: \"SHA1:ee1e52221674da278b60fb7a58c42a2965816b55\"//\"\" | perm: \"WDNVm\"//\"\" | fileid: \"00074387oc0micu9xkf5\"//\"\" | type: CSyncEnums::ItemTypeFile/CSyncEnums::ItemTypeFile/CSyncEnums::ItemTypeFile | e2ee: false/false | e2eeMangledName: \"\"/\"\" | file lock: not locked// | file lock type: \"\"//\"\" | metadata missing: /false/"
[ info nextcloud.gui.folder /run/build/nextcloud-client/src/gui/folder.cpp:641 ]: Ignoring spurious notification for file "$filename$"
[ info nextcloud.gui.application /run/build/nextcloud-client/src/gui/owncloudgui.cpp:251 ]: Sync state changed for folder "$url$" : "Success, some files were ignored."```
Just to throw in an update here. I updated my client to 3.15 and now new problems have arrived. The sync process is really slow and when opening the settings menu or main dialog while syncing, it freezes and i have to click on wait for response several times or force quit the application.
A lot of other people opened bug reports about version 3.15 and I hope they will fix these problems, otherwise the complete product becomes unusable as without the desktop client to work properly and have to set it to "pause sync" the whole day to not burn the SSD, is a major issue for me.
I cross my fingers and hope the best.
just for info, i'm currently using the owncloud desktop client instead of the nextcloud desktop client. I don't think it's a permanent solution, but it works very well so far.
That sounds like a good idea. I definitely will try that out. The current nextcloud client is driving me nuts. As I wrote before, the whole day it is paused and in the evening, when I try to sync it once, that needs about 5 minutes.
I also realized that it deleted some hidden files in my coding projects, like linter configuration and so on. I know its better to use a git service, but these are currently only small learning projects and need no versioning or anything else.
I am still wondering, how such a thing has no priority
I'm currently using the owncloud desktop client as well, it's working without issue for me.
Hey guys, I think I finally solved the problem for me. The nextcloud client now works as expected. The thing I changed is one option in the smb share connection options. The option is called "Check for changes" and can be configured for every smb connection. I now set it to "never", while it was set to "on every direct access" before.
There are still two things I don't know. The first is, why it suddently created a problem, as it constantly synced the share and created a lot of logfiles. The second is, that I still don't really now, for what pupose this is, as when I create a new file locally, it will be synced to the share instantly and works the same way vice versa. So what "never" really means?
Maybe you guys face the same problem with your shares. Check that, otherwise I will close the bug report.
PS: The client still isn't the most stable one, as when I changed the setting and resynced, I got a permission denied error on the client, when deleting a file (while creating a file worked), no matter which direction it synced. So I would recommand to delete the local files and maybe reinstall the client. Change the setting on the smb share and then create the file sync from scratch again.
Thanks and best regards
Thanks, @dishorned . This is the workaround I was looking for and should work until the issue (which seems to be directly related to the setting you changed) is fixed.
The second is, that I still don't really now, for what pupose this is, as when I create a new file locally, it will be synced to the share instantly and works the same way vice versa. So what "never" really means?
I don't think that what you're seeing is supposed to happen. That is, if you make a change to the share locally and the share is set to "Never" check for changes, it shouldn't be propagated to your nextcloud clients. I tested this out by creating a file locally on my share and logged into the account using the share and the file was not present on the dashboard. Previously, it would have been. So, if anyone does implement this workaround and plans on making changes to their share locally, you should run the command /var/www/html/nextcloud/occ files:scan --all periodically so that changes to the local share are discovered.
For reference, the full command I have to run (I have it setup as a cron job) is:
chmod -t /var/www/html/nextcloud/occ && sudo -u apache php --define apc.enable_cli=1 /var/www/html/nextcloud/occ files:scan --all && chmod +t /var/www/html/nextcloud/occ
EDIT: And just to clarify, YES making the change you suggested fixes the issue. However, I still think this is a bug unless proven otherwise so this issue should not be closed.
You are right @bnerickson , when setting this to "never", it does not realize new files created over third party applications or directly on the share. Luckily for me, I am not doing that, so currently no problem for me. But this is only a workaround as you mentioned and not a solution. So i will leave this bug report open.
I am currently using php 8.2 and it's smbclient module. Does anyone runs it on php 8.4? Otherwise I will try this the next days to be sure it's not a problem with the php module.
I think I finally solved the problem for me. The nextcloud client now works as expected. The thing I changed is one option in the smb share connection options. The option is called "Check for changes" and can be configured for every smb connection. I now set it to "never", while it was set to "on every direct access" before.
This workaround also works for external S3 storage. Thank you @dishorned !
Just for everyone who come across this report, I want to mention, that this workaround should be used carefully. If the share is not only used by nextcloud, this can lead to unforseen problems, as nextcloud will not be aware of any new, edited or deleted files though other software.
For this, you can use another workaround as @bnerickson mentioned above, by periodically scan for new files. Another workaround would be, to connect the shares on the unterlying server and use them as local drives, but I haven't tested this.
Hopefully this will be solved soon!
Just for everyone who come across this report, I want to mention, that this workaround should be used carefully. If the share is not only used by nextcloud, this can lead to unforseen problems, as nextcloud will not be aware of any new, edited or deleted files though other software.
For this, you can use another workaround as @bnerickson mentioned above, by periodically scan for new files. Another workaround would be, to connect the shares on the unterlying server and use them as local drives, but I haven't tested this.
Hopefully this will be solved soon!
Mounting Nextcloud as a drive with WebDav does work as an alternative. I did that for a few months before switching to the Owncloud desktop app. On Ubuntu, under "Online Accounts" it actually has an option just for Nextcloud. I still use that for my Calendar and Contacts. WebDav does have limitations that become annoying if you're frequently working with your Nextcloud files.
Perhaps it's because it's so obvious you could've missed it, but no one has mentioned the simplest solution to disk writes. If you don't care about logs, set the app to autolaunch as "nextcloud --logfile /dev/null", which will send them straight to the void.
It's an extreme solution, I'm not going to deny that, but I find it less extreme than nextcloud wearing out my drive with constant writes.
