thunar: copy / paste of directories under mergerfs union mount hangs

[grokskookum]

General description

thunar: copy / paste of directories under mergerfs union mount hangs

Expected behavior

it should complete.

Actual behavior

it hangs.

Precise steps to reproduce the behavior

just about every single time I ctrl x / ctrl v in thunar (especially if the directory exists) the move operation just hangs. this works fine everytime outside of the union mount.

System information

Please provide as much of the following information as possible:

• [ ] mergerfs version: mergerfs -V mergerfs version: 2.29.0 FUSE library version: 2.9.7-mergerfs_2.29.0 fusermount version: 2.9.7-mergerfs_2.29.0 using FUSE kernel interface version 7.31
• [ ] mergerfs settings: cat /etc/fstab or the command line arguments /media/*/mergerfs/ /mergedfs/ fuse.mergerfs defaults,allow_other,direct_io,hard_remove,category.create=mfs,minfreespace=100G,noauto,fsname=mergedfs 0 0
• [ ] Linux version: uname -a Linux workstation 5.3.0-51-generic #44-Ubuntu SMP Wed Apr 22 21:09:44 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
• [ ] Versions of any additional software being used
• [ ] List of drives, filesystems, & sizes: df -h
• [ ] strace of application having problem: strace -f -o /tmp/app.strace.txt <cmd> or strace -f -p <appPID> -o /tmp/app.strace.txt
• [ ] strace of mergerfs while app tried to do it's thing: strace -f -p <mergerfsPID> -o /tmp/mergerfs.strace.txt

[trapexit]

What hangs? Thunar? What happens with regular tools? What distro? Need more info.

[grokskookum]

• the dialogue box for the move operation in thunar just sits there and stops moving even though the files are minuscule.

• yes, Thunar.

• uh.. regurlar tools? well bash commands and scripts work just dandy.

• I am running xubuntu, but it reports as ubuntu: lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 19.10 Release: 19.10 Codename: eoan

• so I recreated the error by executing thunar from the bash prompt (I've used strace as well to try to ID the step that it is spazzing on, but without strace, the last error I see at the bash prompt after executing thunar, is below: "ThunarThumbnailCache: failed to call Copy(): Timeout was reached"

• I have cleared my thumbnail cache, if that matters, which BTW, it does not. the issue persists.

[trapexit]

Can you provide the trace? I was unable to replicate any issues on Linux Mint. I can try in a Xubuntu 19.10 in a VM to see what happens.

[grokskookum]

thunar creates this tempfile, namely ".goutputstream-ITJOK0" in the target directory, it does this first then it does a mv to the proper name to overwrite the file in question, during this hang the tempfile is created but never mv'd. I have a large mv operation that I don't want to interrupt, but I'll create a clean strace when it is done.

[grokskookum]

Does this help any? its part of an strace from thunar: Specifically this line: "2820 ioctl(23, BTRFS_IOC_CLONE or FICLONE, 21) = -1 EOPNOTSUPP (Operation not supported)"

2820  <... flistxattr resumed> NULL, 0) = 0
2820  openat(AT_FDCWD, "/media/starlord/mergedfs/...todo.../todo.out", O_WRONLY|O_CREAT|O_EXCL, 0664 <unfinished ...>
2820  <... openat resumed> )            = -1 EEXIST (File exists)
2820  openat(AT_FDCWD, "/media/starlord/mergedfs/...todo.../todo.out", O_WRONLY|O_CREAT|O_NOFOLLOW, 0664 <unfinished ...>
2820  <... openat resumed> )            = 22
2820  fstat(22,  <unfinished ...>
2820  <... fstat resumed> {st_mode=S_IFREG|0664, st_size=1259, ...}) = 0
2820  openat(AT_FDCWD, "/media/starlord/mergedfs/...todo.../.goutputstream-R80LK0", O_WRONLY|O_CREAT|O_EXCL, 0664 <unfinished ...>
2820  <... openat resumed> )            = 23
2820  close(22)                         = 0
2820  futex(0x7f4b95980f58, FUTEX_WAKE_PRIVATE, 2147483647) = 0
2820  futex(0x7f4b95980f58, FUTEX_WAKE_PRIVATE, 2147483647) = 0
2820  ioctl(23, BTRFS_IOC_CLONE or FICLONE, 21) = -1 EOPNOTSUPP (Operation not supported)
2820  pipe2([22, 24], O_CLOEXEC)        = 0
2820  fcntl(24, F_SETPIPE_SZ, 1048576)  = 1048576
2820  fstat(21, {st_mode=S_IFREG|0664, st_size=1259, ...}) = 0
2820  splice(21, [0], 24, NULL, 1048576, SPLICE_F_MORE) = 1259
2820  splice(22, NULL, 23, [0], 1259, SPLICE_F_MORE <unfinished ...>
2820  <... splice resumed> )            = 1259
2820  write(4, "\1\0\0\0\0\0\0\0", 8 <unfinished ...>
2820  <... write resumed> )             = 8
2820  futex(0x561b40344b90, FUTEX_WAKE_PRIVATE, 1) = 1
2820  futex(0x7f4b88016880, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
2820  <... futex resumed> )             = 0
2820  futex(0x7f4b88016870, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
2820  <... futex resumed> )             = -1 EAGAIN (Resource temporarily unavailable)
2820  futex(0x7f4b88016870, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
2820  <... futex resumed> )             = 0
2820  splice(21, [1259], 24, NULL, 1048576, SPLICE_F_MORE <unfinished ...>

[trapexit]

No. That's fine. That's it just trying to clone the data. Nothing there looks bad/wrong.

[grokskookum]

is there anthing I can grep out of the strace for mergerfs? I would really love to just post what you need not a MB+ file.

[trapexit]

I don't know what the issue is. You're saying it blocks so I would think whatever is around the blocking. There shouldn't be megabytes of data if it stops/fails.

[grokskookum]

mergerfs.strace.txt ok, I redid the test with only a single mount and created subdir-test, and tried to copy / pasted todo into it, then it hung.

[trapexit]

Looks fine. todo directory is created, test file opened, read, has 'yrdy\n' in it, written to new file. Then stops waiting for another request. Need something from thunar.

[grokskookum]

I don't know that to tell you, this test is very simple and it works everytime outside of the union mount and fails almost everytime inside it. (if not every time). I guess I could try nautilus or something. or mc. idk. I really like thunar though.

[grokskookum]

btw, 20 min after the test, I do an lsof and these files are still open.

$ lsof|grep /media/
thunar    15794                         starlord   21w      REG               0,88         5         35 /media/starlord/mergedfs/subdir-test/todo/this.is.a.test...txt
thunar    15794                         starlord   22r      REG               0,88         5         32 /media/starlord/mergedfs/todo/this.is.a.test...txt
thunar    15794                         starlord   25r      REG               0,88         5         32 /media/starlord/mergedfs/todo/this.is.a.test...txt
thunar    15794                         starlord   27w      REG               0,88         5         36 /media/starlord/mergedfs/subdir-test/todo/.goutputstream-AT7OK0
thunar    15794 15795 gmain             starlord   21w      REG               0,88         5         35 /media/starlord/mergedfs/subdir-test/todo/this.is.a.test...txt
thunar    15794 15795 gmain             starlord   22r      REG               0,88         5         32 /media/starlord/mergedfs/todo/this.is.a.test...txt
thunar    15794 15795 gmain             starlord   25r      REG               0,88         5         32 /media/starlord/mergedfs/todo/this.is.a.test...txt
thunar    15794 15795 gmain             starlord   27w      REG               0,88         5         36 /media/starlord/mergedfs/subdir-test/todo/.goutputstream-AT7OK0
thunar    15794 15796 gdbus             starlord   21w      REG               0,88         5         35 /media/starlord/mergedfs/subdir-test/todo/this.is.a.test...txt
thunar    15794 15796 gdbus             starlord   22r      REG               0,88         5         32 /media/starlord/mergedfs/todo/this.is.a.test...txt
thunar    15794 15796 gdbus             starlord   25r      REG               0,88         5         32 /media/starlord/mergedfs/todo/this.is.a.test...txt
thunar    15794 15796 gdbus             starlord   27w      REG               0,88         5         36 /media/starlord/mergedfs/subdir-test/todo/.goutputstream-AT7OK0
thunar    15794 15857 pool-thun         starlord   21w      REG               0,88         5         35 /media/starlord/mergedfs/subdir-test/todo/this.is.a.test...txt
thunar    15794 15857 pool-thun         starlord   22r      REG               0,88         5         32 /media/starlord/mergedfs/todo/this.is.a.test...txt
thunar    15794 15857 pool-thun         starlord   25r      REG               0,88         5         32 /media/starlord/mergedfs/todo/this.is.a.test...txt
thunar    15794 15857 pool-thun         starlord   27w      REG               0,88         5         36 /media/starlord/mergedfs/subdir-test/todo/.goutputstream-AT7OK0
thunar    15794 16144 pool-thun         starlord   21w      REG               0,88         5         35 /media/starlord/mergedfs/subdir-test/todo/this.is.a.test...txt
thunar    15794 16144 pool-thun         starlord   22r      REG               0,88         5         32 /media/starlord/mergedfs/todo/this.is.a.test...txt
thunar    15794 16144 pool-thun         starlord   25r      REG               0,88         5         32 /media/starlord/mergedfs/todo/this.is.a.test...txt
thunar    15794 16144 pool-thun         starlord   27w      REG               0,88         5         36 /media/starlord/mergedfs/subdir-test/todo/.goutputstream-AT7OK0

[trapexit]

I just installed Xubuntu 19.10 in a VM, updated it, installed mergerfs, ran mergerfs -f -o allow_other,use_ino,direct_io,category.create=mfs /home/user /tmp/mnt, copy and pasted /user/lib into /tmp/mnt/Documents. Copied nearly 3GB fine.

How did you install mergerfs?

[grokskookum]

I used a .deb.

-rw-r--r--  1 root     root     223940 Feb 24 11:13 mergerfs_2.29.0.ubuntu-bionic_amd64.deb
# dpkg -l |grep mergerfs
ii  mergerfs                                                    2.29.0~ubuntu-bionic                   amd64        another 

[trapexit]

It shouldn't matter but you're using eoan. That's what I installed. I can retest with the bionic build.

[trapexit]

No dice. Both work fine when copying data around through Thunar. Do you have another machine or VM you could try to reproduce this issue? Have you tried other file managers or programs?

[grokskookum]

I'll try some stuff. I have a feeling whatever issue I am having has to do with gvfsd_metadata or gdbus, but that's just a useless gut feeling.

Thanks for working with me on this.

[trapexit]

No problem. I've been poking around for about an hour now and can't get it to hiccup (outside when I didn't give the VM enough RAM and it started swapping.) If you can reproduce it with other apps that might help or be easier to trace. Unfortunately, I doubt it'd be easy to get a useful stack trace without a debug build.

[grokskookum]

• BTW, nautilus also acts broken under the union mount in similar ways. if I copy a directory and paste it over another with the same name it only merges the first filename into it and then creates the same garbage temp file that thunar created with a name like .goutputstream-KJ2NK0. it doesn't hang... the dialogue box completes but the copy is definitely not correct or complete.

• Nautilus acts perfectly on this merge copy outside of the union mount, same as thunar. but broken inside /media/starlord/mergedfs/, same as thunar.

• Basically to recreate, I make a temp dir, put some text files in it, copy it to another directory then merge it back on top of the original, with or without more files, it is always broken under the union mount with thunar or nautilus. my merge copy bash scripts don't use those weird .goutputstream temp files so it is unaffected.

I'll keep hacking at it.

edit: to be clear, sometimes a simple directory copy paste will hang with the same uncancellable dialog box, without attempting a merge.

[trapexit]

What do you mean "merge it back on top of the original"? What paths are we talking about?

[grokskookum]

So, one reason I like thunar is that if you have a directory:

/media/starlord/mergedfs/this.is.a.directory/ # and this directory has a dozen files in it

and you have another directory:

/media/starlord/mergedfs/sorted.by.prefix/this.../this.is.a.directory/

you can cut / paste the first one on top of another directory with the same name but located elsewhere and thunar will merge the 2 directories and ask you if filenames collide if you want to keep the existing files or overwrite. nautilus used to actually copy each file and it took much longer while thunar did the operation much faster with presumably mv operations.

if I attempt to merge 2 directories with the same name under the union mount it pretty much always gives me an uncancellable unmoving dialogue box even though their might be only 1 file and its 1 byte long. I can hit the little x on the top right of the dialogue box but every single time I try to move anything with thunar cut / paste it will bring it back up like its been in the background the whole time. the only way to get rid of it is to killall thunar Thunar. nautilus seems to be doing the same sort of thing but without the hung dialogue box. (it creates the junk temp file and only moves a single file, even if the source dir had dozens.)

I can use a bash script to merge directories just fine, but it of course doesn't use any backend stuff like gvfsd or gdbus or whatever and no temp files.

[grokskookum]

I edited my last comment, I keep saying copy / paste and I mean cut / paste (ctrl-x / ctrl-v), but either create the issue.

[grokskookum]

I tried another file manager, double commander works fine under the union mount. so it seems to me its something to do with whatever nautilus / thunar use as a backend. perhaps the gvfsd-* processes.

[trapexit]

What happens in Thunar if you just copy and paste a random file into a random directory?

[grokskookum]

Usually it works, but often (like just now) it fails the same way, again this is just a copy paste of a single file. no merging or anything weird. (I did it a few times just to be dramatic and it failed every time.) and this never goes away. In an hour it will still be there.

[trapexit]

Maybe try an older release? Run mergerfs in debug mode? Without being able to reproduce it in any form I'm kinda stuck.

[grokskookum]

I think I'll upgrade my box to 20.04, and see if it follows me, and then see if I can reproduce it in a vm.

[trapexit]

Any updates on this?