[Bug]: Some virtual files cannot be sync'd: "Conflict when uploading a file. It's going to get removed!"

[wonx]

trafficstars

⚠️ Before submitting, please verify the following: ⚠️

• [X] This is a bug, not a question or a configuration issue.
• [X] This issue is not already reported on Github (I've searched it).
• [X] Nextcloud Server and Desktop Client are up to date. See Server Maintenance and Release Schedule and Desktop Releases for supported versions.
• [X] I agree to follow Nextcloud's Code of Conduct

Bug description

After having uploaded a few thousand JPG images from another computer (Windows 10) without issue, when these changes have been synchronized to my Linux laptop (Kubuntu 22.04) with virtual files enabled, all the files in some specific subdirectories disappearing after the client (v 3.5.1), while showing the message "Conflict when uploading a file. It's going to get removed!" for each one of them. See attached screenshot:

As I mentioned, only files in a few subdirectories have disappeared, but no from others. I can't find a reasonable pattern.

These files have only been deleted from the local machine, but remain in the server and in the Windows PC, so I believe it's an issue linked with the desktop client in the linux machine.

One more thing, these files are stored in an external storage and accessed via SMB.

Steps to reproduce

1. Upload a directory tree with files in one computer. Wait until synchronization finishes successfully.
2. Turn on second computer with linux and desktop client 3.5.1. Wait until synchronization finishes too.
3. Observe how some directories trigger this problem and many files disappear.

Expected behavior

All files from the server should appear in the client directory tree.

Which files are affected by this bug

No clear pattern. All affected files were .JPG

Operating system

Linux

Which version of the operating system you are running.

Kubuntu 22.04

Package

Distro package manager

Nextcloud Server version

23.0.2

Nextcloud Desktop Client version

3.5.1

Is this bug present after an update or on a fresh install?

Updated from a minor version (ex. 3.4.2 to 3.4.4)

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

Are you using an external user-backend?

• [ ] Default internal user-backend
• [ ] LDAP/ Active Directory
• [ ] SSO - SAML
• [ ] Other

Nextcloud Server logs

No relevant logs in the server. The issue seems to be exclusively from the side of the client.

Additional info

(I can provide the Debug Archive log if necessary, but the file is more than 500MB in size and contains lots of sensitive information)

[wonx]

I have also seen the same error today, but when renaming files. After the files in a folder have been renamed, they returned to their previous name. If I try to rename them again, the same happens. However, that not happened on other almost identical folders in the same path.

[pasbec]

I ran into a similar massive bug today on Windows with the latest client 3.5.1. Not sure though what caused the problem. The client went totally crazy and deleted one of the first level folders with thousands of files and 400 GB in total. The server trash was completely empty afterwards. Without manual backups everything would be gone permanently now. Trying to disable virtual files immediately crashed the client machine with a blue screen. The deleted stuff was however still visible as "cloud only" on the client after the deletion. But trying to access or download one of these files/folders gave an "cloud operation not successful" error. After reboot, I was able to remove my account from the client and now the restore process will be running for the next two days.

This virtual file system stuff is dangereously buggy! Never going to use this again... .

[nihues]

I also have this problem, some users are reporting this issue on my nc, I'm not sure what is going on, no logs apparently, it just send this message and auto delete the files.

Using Nextcloud Desktop 3.5.2 (server is 24.0.2) but happens some months already (or at least since that file timestamp bug that removed many files).

It happens also when creating folders too, it creates a folder, then trigger this and auto delete the folder, sometimes creating new folders get same result, sometimes not, changing name of created folder stop the bahavior, sometimes not, or creating outside the folder tree.

I can find the files in trash and recover in most times.

I'll try to find some info on log if triggers again and post here.

[AndyXheli]

Same issue i think. I created this ticket a while back and bot closed it ;( https://github.com/nextcloud/desktop/issues/4269

[AndyXheli]

I just lost one of my most important programs. They really need to fix this. I don't understand how this ticket is not getting any feedback from dev. Come on i had ticket #4269 open since Fed 11 and nothing from dev also another ticket #3912 The VFS is defiantly not stable. Why is the client deleting my data? without my approval? This should not happen.

[nihues]

You should delete that image... it's not really yours by the name of it... otherwise just grab from your Adobe Account as you had bought it, isn't?

Moving to the topic, can't really find any info on this on the logs, if I can find I'll post here.

[kwisatz]

Confirming that I've seen (am seeing) a similar behavior on Windows. Lost 2 hours worth of work due to this :(

I created a file at 16:30, then modified it until 17:10, the file never showed a check-mark icon and when it finally did, it was back to the 16:30 version.

Windows notifications are full of Conflict when uploading a folder. It's going to get removed! and this seems never ending. On top of that, the Desktop client keeps crashing. Settings are inaccessible apart from the very first couple of seconds after Windows start. VFS to me is clearly not production ready.

Update

I was able to recover the file by stopping sync, disabling virtual file support, and re-synching. It then showed me unresolved conflicts for this file:

[ItsMeRitch]

Can confirm I’m also seeing this issue with my Raspian 32bit build. I find that “removed conflicts” are sent to my trash bin thankfully, but in the past, with the trash disabled, it’s gone for good.

[infibility]

I also faced same problem with V3.5.4 on windows11. I moved virturl folder to another path which contains many jpg files. when processing, I opened some virtual file and the sync has stopped. after download and open the virtual file, rename path resumes and the error occured. my files are permanently gone... ( and I need to recorver from backup)

[pasbec]

The part which causes all the trouble is in libsync, more specifically in a lambda function in discovery.cpp. Corresponding translations of the error message are here: e.g de, ...

The place where things get deleted (why???) should only be reached for new files and renames, which perfectly matches the description of this issue.

Furthermore there is some note in the code

// TODO: We may want to execute the same logic for non-VFS mode, as, moving/renaming the same folder by 2 or more clients at the same time is not possible in Web UI.
// Keeping it like this (for VFS files and folders only) just to fix a user issue.

This also explains why this happens only if VFS is used. And there is some user issue mentioned.

The whole stuff got "recently" changed with pull request 3449 by @allexzander and @mgallien. I don't know how the Nextcloud Client works in detail, but it could be that the symptoms of the present issue is some sort of regression of those changes or likely in combination with yet another change.

I'd really appreciate if either @allexzander or @mgallien could comment on this.

[allexzander]

I also faced same problem with V3.5.4 on windows11. I moved virturl folder to another path which contains many jpg files. when processing, I opened some virtual file and the sync has stopped. after download and open the virtual file, rename path resumes and the error occured. my files are permanently gone... ( and I need to recorver from backup)

Any chance you could come up with steps that always trigger the issue?

[infibility]

I'm sorry for late. not always, I think so I can't tell the bug reproduces but just one of possibility.

[AndyXheli]

@allexzander the way ive seen this issues come up with me is if i have a folder with alot of files and i drag and drop them into the nc dir and files are gone. I just did it right now and all files got removed

[AndyXheli]

@allexzander I was able to replate it and I got some logs for you please see attached Log.txt

[AndyXheli]

Hi @allexzander any updates on this from your team?

[nihues]

Yesterday I've moved on windows explorer one folder inside another (double click mouse madness) then I've tried to move back the folder to the right place after a few seconds and got all files with same error (conflict it will be deleted thing).

So | /originalfolder -> moved to /wrong/originalfolder (all fine no error). Seconds later (not synced yet) | moved /wrong/originalfoder --> /originalfolder (got error and ALL files disappeared from both folders (even the folders went caput), there was 1 or 2 files on the /wrong/originalfolder that probably was already synced and stayed there.

No files in trash but after investigating found them in /originalfolder (on the server and missing from nc desktop or web cloud) so a occ files:scan for the folder worked and all went back.

NC Desktop: 3.6.0 NC Server: 24.0.4

Logs from the moment: 20220922_1139_owncloud.log.0.gz 20220922_1113_owncloud.log.0.gz 20220922_1138_owncloud.log.0.gz

[adripo]

I just had the same issue. Is there any workaround to recover removed files? ⚠️⚠️⚠️ I think this issue should be prioritized accordingly because it deletes files irreversibly without any prompt.

[AndyXheli]

Yes, i would agree. Maybe an option once the desktop client detects that there's a conflict and its going to get removed there's a user prompt to keep it and try to resync the files instated of removing them

Maybe something like Keep & Resync or Remove button

[adripo]

@claucambra @camilasan can we have an update on this issue please?

[rgreeott]

I have experienced the same issue (with several recent versions of the sync client) and NC 24.0.5. The server resides on EC2 with external Azure filestore and S3 attached. At any given time there's a minimum of 32 virtual sync clients, up to 50, attached. There's ~ 2.5T, mostly images and drone footage totaling ~ 700K files. If a mistake is made while files are uploaded and the error corrected (ie. renaming or moving files), the offending client errors with "conflict when uploading a file. It's going to be removed". All other client error with sync conflicts. Depending on the HW resources of the client machine, it can render the system useless. In many cases, the sync client crashes/resets and forces a re-scan, which places enormous load on the server. In the case where files have been deleted from Azure, the DB records still exist. Thus, various clients and web interface still displays references, as if the files still exist.

[FesTGig]

I have a similar issue. NC 25.0.1 / Client version 3.6.2 on macOS, virtual files enabled. After moving files on the web interface, the client tries to sync them back to the local mac and errors with "conflict when uploading a file. It's going to be removed". The process take huge load on the local machine and crashes the client. After a day watching this procedure I stoped the synchronisation process and hope for a solution. Are there any news on this issue?

[cool-clouds]

Same issue on windows desktop client 3.6.X. Server : Ubuntu 20.04 NC version : 24.0.8

[Luxamman]

I've had the problem minimum twice (also today), but in Windows and without virtual files.

• Windows 11 22H2 - 22621.2215
• Client 3.9.4
• Nextloud 27.0.2
• enough storage space available
• Also found in #3912 i think.

-- I uploaded a folder with a larger video (~700MB) when the video file was simply gone. The folder is still there. The file is not in the trash either. Simply deleted without question or notice.

Translation of the screenshot, from bottom up:

"You created whateverfolder." (the Folder) "whateverfile.mp4 synced." (marked with red X) "You have deleted whateverfile.mp4." (No I haven't, marked with red X) "File upload conflict. It will be removed!"

--

But it also says: "You deleted whateverfile.mp4". Why that?

And I think that's very problematic because then the files are just gone. I didn't copy it, moved it in. So, as I said, the file is simply gone, because it wasn't a copy. This shouldn't happen and is very annoying. Why can the client even completely delete files? Not even in the trash? What?

Thanks in advance for looking into this and helping!

[Killea]

this happened when I cut files from another folder. Client Version 3.10.0 (Windows) event there's a conflict, it should add a prefix or rename the file. Delete the file is a big wrong.

[gersur]

LOL WTF, I JUST SPEND 3 HOURS TO SCAN DOCUMENTS AND I MOVE TO A SUBFOLDER, ALL MY FILES ARE GONE

[allexzander]

So, I can see the root cause when performing a move of a folder with many files while the previous move has not been completed yet. Here is what happens:

• remote move happens with one API call - just one folder gets moved and all nested items are moved automatically
• after the remote move is done, each item inside the move folder has to be updated in the local database and it requires N * SQL queries, where N is the number of all nested items of a folder recursively, during that update - a new path, pathlen, and phash are set, and phash is calculated based on new path and pathlen is set to a new path length
• in the case of VFS placeholders (when Virtual Files is enabled) - each placeholder needs an update on the disk (its metadata is getting updated)

Possible solutions:

• update all nested items recursively with one SQL query (need to figure out how to update calculated values like phash and pathlen even if we can update path by simply replacing the prefix with a new location of a parent, these values need recalculation, if one query is joined from smaller queries to update single row - it will be very slow)
• lock the local sync folder for modification via OS means programmatically, such that new moves won't be possible while there is a move in progress and db records are being updated
• what else?

In a nutshell, possible solutions are either to try to update the local database for all the nested items (there can be thousands and more files) in one SQL query, or, limit the freedom of multiple moves (renames) such that the user can not mess with folder locations until the move has completed.

Correct me if I am wrong, but after studying all replies, I am coming to a single use-case for this issue (that I can reproduce) - moving a folder with many files multiple times while the previous sync is still running.

Workaround: Not really a workaround, but, a recommendation until this is fixed - to not move folders too quickly, while the previous move is not finalized yet.

For the future: I also have the idea that setting file records to the local database needs optimization such that if we are syncing with VFS mode for the first time we would not spend too much time inserting every record into the database, which would speed up the initial sync as files are not downloaded to disk. This, however, might become an issue if the client is killed/crashed in the middle of such sync.

@mgallien @camilasan @claucambra Any other ideas?

[allexzander]

Not yet in a 100% working state (placeholder can't be hydrated without client restart after move, possibly need to handle placeholder update after move differently with new database logic code). The idea with mass update of all nested items recursively works with custom SQL function and UPDATE query https://github.com/nextcloud/desktop/tree/feature/single-sql-query-on-move. Move is now really fast to reflect in the database (a 1000 files is move recorded in just 3-5 seconds instead of moving them one by one that takes X amount of time depending of number of items).

[allexzander]

As for the scenario of a new folder just moved from a non-Nextcloud folder, I could not reproduce it. But I have an idea of how to prevent this from happening. Now we create file records in the local db only after a new file/folder gets uploaded with success from the server. Maybe we can change this, and first create a db record like we already know about the new directory and all its items we just moved to the Nextcloud sync folder, and introduce a state column in the DB, like "LocalNew", or "LocalNewPendingUpload", then, if anything happens - like an error on the server or Cf API error, we will have a record in the local DB it will prevent corresponding files/folders from removal? @mgallien what do you think? This might require quite some changes to the sync engine and database.

[lildadou]

I've done a few tests and I think that binary content is the most decisive factor:

Cases that do not cause the bug:

• copy-paste a 140Mio zip file
• copy-paste a 6Gio ISO file
• copy-paste a folder containing a 140Mio zip file and a 6Gio ISO file
• copy-paste a 275Mio video file
• copy-paste a folder containing 3 x 275Mio video files
• drag-and-drop 140Mio zip file
• drag-and-drop 6Gb ISO file
• drag-and-drop a folder containing a 140Mio zip file and a 6Gio ISO file
• cut-and-paste 140Mio zip file
• cut-and-paste 6Gb ISO file
• cut-and-paste a folder containing a 140Mio zip file and a 6Gio ISO file
• zip a folder containing 3 x 275Mio video files then cut-and-paste the zip file
• rename a 140Mio zip file as a video file then cut-and-paste the renamed zip file

Cases that do cause the bug:

• drag-and-drop a 275Mio video file
• drag-and-drop a folder containing 3 x 275Mio video files
• cut-and-paste a 275Mio video file
• cut-and-paste a folder containing 3 x 275Mio video files
• rename a 275Mio video files as a zip file then cut-and-paste the renamed video file

The last test case is very disturbing.

Of course, there's no special added server-side processing (such as anti-virus), all the files in the test case were in the same sourced folder and moved/copied to the same Nextcloud virtual folder.

Windows Desktop Client 3.12.1

Edit: 302Mio zip file works too

[platima]

Yeah that makes sense - it seems that most of the files I've lost were binary files and almost always around a file or folder rename.

That being said, I have had some issues with plain text files too, but that is more around versioning it seems, and specifically with the 'Notes' app, so might not be something to do with the desktop sync, although they do get stored in a synced folder and renamed from time to time.

[infibility]

Hello I happend to face this situation again. I just remamed the folder that contain 60GB photo and Videos. thsi folfer is not fully downloaded to local. after that, nextcloud client start to upload the downloaded files to renamed folder. Client did`t recognize as rename but delete old folder and start to upload to new folder, with error message "Conflict when uploading a file. It's going to get removed!". After finishing the sync, the folder is only 29GB. 31GB of data is gone. I am using nc client version 3.12.3 on Windows 11.

[platima]

This is really bad. The software is literally erasing peoples data without even asking or taking a backup.

This ticket should be treated as a priority ASAP.