Improve inter-qube file copy/move behavior when files conflict

[kirill9000]

The problem you're addressing (if any)

When copying or moving files between qubes, the operation will fail if a file with the same name and path already exists in the destination qube. This currently requires the user to first manually delete or rename the conflicting file.

This UX problem does not exist on conventional OSes or within a single qube, because conventional OSes usually prompt the user to rename, overwrite, or skip one of the conflicting files. The problem described here is specific to the way Qubes OS currently handles inter-qube file copy/move operations, because Qubes OS currently does not have a way to handle such file conflicts.

The solution you'd like

Improve the UX of this situation in whatever ways the developers deem best.

A few possible ideas offered by users:

• Copy/move at least the non-conflicting files instead of failing the whole operation.
• Prompt user to rename or delete conflicting files.
• Automatically rename conflicting files.

(See comments below for more.)

The value to a user and who that user might be

Better UX for users who encounter this situation.

Completion criteria checklist

(This section is for developer use only. Please do not modify it.)

Original issue description

qvm-copy-to-vm fails if some file already exists on destination VM. But when copying a directory, I think it would be better if it transferred files that are not present on destination VM (while skipping already existing of course) rather than fail.

[bnvk]

I think it would be better if it transferred files that are not present on destination VM (while skipping already existing of course) rather than fail.

This does sound like an improvement, IMO but unsure about feasibility. Hey @marmarek is doing this sort of thing possible?

[marmarek]

This does sound like an improvement, IMO but unsure about feasibility. Hey @marmarek is doing this sort of thing possible?

Technically it is possible (quite trivial). But I'm afraid of some reporting here. For example if user have some document he/she want to send via email, it requires copying it into email VM. If there was already file identically named (for example its previous version), such skipping instead of failing will most likely mean the user will send old version of the document without noticing it.

[andrewdavidwong]

If there was already file identically named (for example its previous version), such skipping instead of failing will most likely mean the user will send old version of the document without noticing it.

Very good point. Expected user result would probably be something like:

1. file.txt already exists in destvm
2. [user@sourcevm]$ qvm-copy-to-vm destvm file.txt
3. file.txt and file (from sourcevm).txt now both exist in destvm

This would mirror the way most OSes handle copy/pasting a file into the same folder as the original. Usually you end up with the original file.txt and the copy is named file (1).txt or copy of file.txt. (But of course qvm-copy-to-vm might deliver a file with the same name but different content, so we don't want to just assume it's a "copy.")

[marmarek]

The (from sourcevm) addition doesn't scale well - what if that file exists too? Maybe simply use numbers: file.txt, file (1).txt, file (2).txt etc. But I'm still not so sure is it a good idea at all. Sooner or later it will lead to using wrong file. Maybe better have some GUI window asking to skip/replace the file in case of conflict (with default of skip)?

[andrewdavidwong]

The (from sourcevm) addition doesn't scale well - what if that file exists too?

Well, I figured it would be something like:

file.txt
file (from sourcevm).txt
file (from sourcevm 1).txt
file (from sourcevm 2).txt
file (from sourcevm 3).txt

This would be in keeping with the way most conventional system seem to handle renaming copies. Personally, I dislike spaces and parentheses in filenames.

But I'm still not so sure is it a good idea at all. Sooner or later it will lead to using wrong file.

At some point it, it has to become the user's responsibility to choose the correct file. If the user wants to create n copies of a file, then the user should expect to have to choose from among n roughly homonymous files.

Maybe better have some GUI window asking to skip/replace the file in case of conflict (with default of skip)?

That might be better. It would make the choice more explicit to the user and alert the user to the fact that there already exists a homonymous file at the destination. However, it would also add to the (mental and physical) overhead of performing the copy operation.

[brendanhoar]

Rather than open a new issue and incur the (appropriate) maintenance wrath of andrew, I searched, found this issue and have a similar request, but want to point to the mv commands option and suggest we follow mv's lead on this. So here is the draft version of the issue report I was about to give:

Under R4.0.1, the qvm-move command can commonly fail in a somewhat painful to review and recover from manner.

Example:

cd /tmp
mkdir qubes-issue-qvm-move-test
cd qubes-issue-qvm-move-test
touch file1 **file2** file3 file4 file5 file6 file7 file8
qvm-move file1 **file2** file3 file4
# [user chooses a valid VM and clicks OK]
touch **file2**
qvm-move file5 file6 **file2** file8
# [user chooses same valid VM and clicks OK]

The first qvm-move command works. Second qvm-move command: a) copies file5 and file6 to target, b) errors out part way c) keeps both copied and uncopied files in source, nothing is deleted in source as part of the move

I would prefer, the following behavior: The second qvm-move fails with no files copied or moved (no side effects) because the target VM would pre-validate lack of name collisions at the first level in /home/user/QubesIncoming/$VMName/ (note: while this allows the source VM to know something about files in the target VM, this knowledge already is passed over in the currently implemented behavior of partial failure, perhaps at finer granularity). Notably, it only needs to pre-validate no collisions of the filenames and directorynames at the top of the tree being moved.* -and- 2. A new CLI option would be available (similar to the mv command's --backup=number but perhaps less wordy) the user could provide that would tell the target VM to algorithmically rename the target file to prevent erroring out at all.

B

• I will handwave about changes to the target fs after the namespace collision check is performed but during the copy/move for now. I suspect that occurs much less often.

[marmarek]

target VM would pre-validate lack of name collisions at the first level in /home/user/QubesIncoming/$VMName/

pre-validation is prone to race conditions - if the conflicting file is created during the operation (for example two concurrent qvm-copy/qvm-move operations). Alternative with similar effect would be removing already copied files if of them failed. Not the best option from performance point of view (imagine sending directory with hundreds of files and the last one hit conflict), but since this should be rare case it may be acceptable.

A new CLI option would be available (similar to the mv command's --backup=number but perhaps less wordy) the user could provide that would tell the target VM to algorithmically rename the target file to prevent erroring out at all.

This behavior definitely should not be controlled by the sending domain. If anything, then by receiving one (some config option or such). Also, preventing overriding files is on purpose. This include the case when old version is moved aside. Otherwise sending domain could override the file just before you want to use it (but after initial validation the file content, including any signature checks etc). But see above about alternative options - basically saving new (conflicting) file under another name.

[brendanhoar]

Thanks for the feedback!

I do appreciate the correction to my understanding of mv --backup (you're right that it renames the existing file, not the new file, which could be more problematic than I'd considered).

Stepping back, my summary position is: don't leave the copy/move in a state of partial completion requiring a lot of fiddly and error prone review by the user in nearly as many cases as it does now.

One can imagine several mechanisms to reduce/prevent collisions:

1. Target-controlled file-level renaming semantics.
2. Target-controlled copying into unique directories per transaction, e.g. date/time stamped directories such as ~/QubesIncoming/work/20190209123233 (granularity at one second or finer, with target-controlled auto increment on colliding times), lock held on dir by target's receiving process forcing parallel receiving processes with time collision to have to create their own receiving directories with incremented times.
3. Variant of #2 which treats the owned directory as temporary and locally attempts to merge the contents up a level after all data is received (if successful, temp dir is removed), perhaps with or without local renaming rules (avoiding the source having control, as you said). Has similar issues as #1, but separates out renaming considerations until after all data is present.
4. ...

Thanks for listening.

Brendan

[donob4n]

pre-validation is prone to race conditions - if the conflicting file is created during the operation (for example two concurrent qvm-copy/qvm-move operations). Alternative with similar effect would be removing already copied files if of them failed. Not the best option from performance point of view (imagine sending directory with hundreds of files and the last one hit conflict), but since this should be rare case it may be acceptable.

Could multiple copy/move operations be queued instead of run concurrent? It will avoid race conditions errors, fix https://github.com/QubesOS/qubes-issues/issues/5950 and probably get better hard disk performance.

[brendanhoar]

Moving and updating my comments from #6406.

Since my last post to this thread, I believe I’ve internalized Marek’s critique of my simple mv-like solution from the last conversation. I’ve looked a bit more into dom0 policy configs and how they might be utilized, along with a dom0 dialog.

I now propose that dom0 should send a behavior parameter to the file receiver part of the client in target VMs. The parameter should indicate what behavior the receiver should perform. The parameter is generated in dom0 based on dom0 controlled policy and/or dom0 contained UI choices added to the target VM dialog that already exists.

In particular, I propose several of the below possible behaviors as candidates and controlled in the target VM by the dom0-sent parameter.

These are behaviors users might want (though perhaps some would likely fall out after security or LOE analysis). The target VM receiver would send notifications back to dom0 indicating success or what type of failure(s) were encountered: file name conflict, permissions issue, filesystem full, etc.). dom0 may also have an additional policy indicating what level of information should be sent to source VM (perhaps defaulting to current behavior of either success or generic error).

The list of potential target VM behaviors controlled by dom0 (all report result back to dom0)...

Current behavior (default?): on filename conflict or other error, leave already copied files in place.

Cleaner version of above behavior: on filename conflict or other error, rollback written files.

Rename behavior # 1: on filename conflict, rename old file (include cascade renames if necessary) and continue. On any other errors rollback everything, including in renaming older file. I could see this fall out for security reasons, but I also see utility for source code syncing.

Rename behavior # 2: on filename conflict, rename new file (include cascade rename if necessary) and continue. On any other errors rollback everything.

Overwrite: on filename conflict, overwrite target file (implemented by temporarily renaming old file and removing only after successful receipt of all files). On any other errors. rollback everything. This seems riskiest but I could see some use cases.

To reiterate, the flag sent to the target VM would not be controlled by source VM but instead via a combination of Qubes policy in dom0 and new UI components added to the target selection dialog box in dom0.

Implementation would require some additional changes beyond the above to the file receiver, such as possibly sending all incoming files to a hidden directory first and then applying policy to merge/rename/overwrite/rollback.

Under default file transit notification policy, source VM would never know the reason for failure, just like today. One could investigate whether a broader set of policy choices is warranted, but I’ve written too much already.

brendan

[ninavizz]

Regarding a proposal of mine to automatically rename duplicate files with sequential numbered addenda in #5853, @marmarta said:

@ninavizz this shouldn't be a default, though. Because if this happens silently, without informing user very well what's happening, you can easily get a problem like "you copied a file to attach it to an email, and everything went well, no errors, but if you attach the file you thought you copied, you'll get an old version" (because the new version is actually called differently).

Below are screenshots from my Mac. Firefox does the parenthetical numbered files, when I save duplicates of the same downloaded file; my design software, does the hyphenated parenthetical. I pinged the FPF Slack to see how it's done on Windows and Linux machines, and learned that each application determines how the renaming is done—and that in fact, numerical renaming is a standard pattern across platforms. Just, as per the below screenshot, sometimes different per application.

While, yes, if a user is browsing for files by date when looking to attach something a-la email, they would in fact miss that a newer one with a numbered addenda also existed in that same location. Unfortunately, because the numerical renaming schema feels like a very standard pattern, I'd prefer to let that remain a "problem" as Marta spoke to it as.

Not all design problems have 100% perfect solutions—and this is one of those. A "more perfect" solution, would likely be prompting the user to rename a file, anytime there is a conflict. However, that could also be received by the user as highly annoying.

2x unfortunately, because I am a mere non-technical mortal, I do not understand a word of @brendanhoar's wonderfully thought-out technical solution, described above.

May I cast a vote for a numerical renaming schema, similar to what Firefox does (with the parenthetical)?

[andrewdavidwong]

Below are screenshots from my Mac. Firefox does the parenthetical numbered files, when I save duplicates of the same downloaded file; my design software, does the hyphenated parenthetical. I pinged the FPF Slack to see how it's done on Windows and Linux machines, and learned that each application determines how the renaming is done—and that in fact, numerical renaming is a standard pattern across platforms. Just, as per the below screenshot, sometimes different per application.

On Windows and Linux, in my experience, the standard behavior for downloading the same file in a browser multiple times is to add numbers, but the standard behavior when copying, moving, or saving a new file to a folder that already has a file with the same name is to ask the user whether they want to overwrite the file in the destination folder.

[ninavizz]

On Windows and Linux, in my experience, the standard behavior for downloading the same file in a browser multiple times is to add numbers, but the standard behavior when copying, moving, or saving a new file to a folder that already has a file with the same name is to ask the user whether they want to overwrite the file in the destination folder.

Keeping in mind that I cannot understand most of what the wonderful and @brendanhoar thoughtfully composed above—and just to confirm things before I invest any time in a wireframe—were we to go with the above, the experience would look like...

• I, the user, select "copy" or "move" the file myfile.add from my right-click menu in the File Manager, from qube-a
• I elect to move the file to qube-b which is not yet open
• Notifications fly about, and qube-b opens

Where I am on the fence, is that in a standard move/copy action, the File Manager is not opened in the destination qube. It all "just happens." For a rename action, it would be weird for the File Manager to just go and open on its own, and then ask the user to rename the file.

So: Which mechanism should show an initial message, telling the user "myfile.add" already exists—would you like to rename or overwrite the OG file—an OS level notification that would open the file manager if the user chooses "rename," or... ???

[brendanhoar]

I was confusing for several reasons (I'm sure), but one of the reasons is this: generally, the Qubes (team/OS) default policy is that a qube should not leak information about its contents/existence to another qube without explicit consent of the user.

Notably the default dialog box presented to the user (where target is chosen), when copying/moving files, is not a dialog box in the source qube, but in dom0.

So the way I wrote my suggestions above had to do with not informing the source qube about the existence of conflicting filenames in a/the target qube.

Rather rename/overwrite/abort behavior should be determined by policy configuration (managed in dom0), which could also optionally include a dom0 dialog that the user interacts with if the policy includes "ask".

The source qube should only know if the data transfer operation failed or succeeded (particularly for "move" operations).

[ninavizz]

...good thing I know someone who's also working on Policies stuff, r/n. TY @brendanhoar for that helpful explanation! :)

[brendanhoar]

Heh, good.

I'm glad the community has the usability-focused folks and security-focused folks working so closely.

And as I'm just a dilettante...I'm also glad you can bounce what I've said above off the folks who actually know their stuff . :)

[andrewdavidwong]

On Windows and Linux, in my experience, the standard behavior for downloading the same file in a browser multiple times is to add numbers, but the standard behavior when copying, moving, or saving a new file to a folder that already has a file with the same name is to ask the user whether they want to overwrite the file in the destination folder.

Keeping in mind that I cannot understand most of what the wonderful and @brendanhoar thoughtfully composed above—and just to confirm things before I invest any time in a wireframe—were we to go with the above, the experience would look like...

• I, the user, select "copy" or "move" the file myfile.add from my right-click menu in the File Manager, from qube-a
• I elect to move the file to qube-b which is not yet open
• Notifications fly about, and qube-b opens

Where I am on the fence, is that in a standard move/copy action, the File Manager is not opened in the destination qube. It all "just happens." For a rename action, it would be weird for the File Manager to just go and open on its own, and then ask the user to rename the file.

So: Which mechanism should show an initial message, telling the user "myfile.add" already exists—would you like to rename or overwrite the OG file—an OS level notification that would open the file manager if the user chooses "rename," or... ???

I didn't mention renaming or the system automatically opening file manager windows at all. The standard workflows I described don't involve either of those things. There is simply a dialog box that informs the user that a file with that name already exists and asks the user whether they want to overwrite the existing file.

This already happens in Qubes OS, just without prompting the user to overwrite. When I try to copy a file from one qube to another, and a file with the same name already exists at the destination, I get this error message:

qfile-agent: Fatal error: A file named “foo.txt” already exists in QubesIncoming dir (error type: File exists)

The message window is from the originating qube, not dom0.

[mfc]

hi all! i find this annoying, almost daily :)

thanks andrew to reframing this issue to make some of the proposed ideas clear ( i previously put some ideas in https://github.com/QubesOS/qubes-issues/issues/9987 ), basically there should be the same dialog as when a user is transferring a file to someplace in the same qube that already has a file named the same --- the user is given a dialog asking if they want to replace the destination file, rename the incoming file, or cancel transfer.

my assumption would be this process/dialog is controlled by dom0.

would be great to reduce Qubes-specific user frictions where we find them :)