moveonenospc causes severe file system slowdown when a large file has to be moved

[Devonavar]

General description

moveonenospc causes severe file system slowdown when a large file has to be moved. When the file is large, the merged filesystem can become nearly inaccessible for long periods of time while the copy is happening, likely due to disk thrashing. This can lead to the file create timing out.

Expected behavior

moveonenospc should be fairly transparent to the user; an attempt to access other resources on the file system should not freeze for multiple minutes while moveonenospc completes. Some performance hit is expected and acceptable, but performance should not be significantly worse than ordinary disk thrashing.

Also, writes should not fail because of moveonenospc.

Actual behavior

When this situation occurs, performance seems to be worse than regular disk thrashing due to multiple files being accessed via the share. Ordinary thrashing merely slows file access down (sometimes significantly, but not unusably). When moveonenospc is triggered, file access becomes so slow that the file system can appear frozen for multiple minutes.

Precise steps to reproduce the behavior

Explicitly list all steps to reproduce. Preferably create a minimal example of the problem using standard command line tools. The more variables (apps, settings, etc.) that are involved the more difficult it is to debug. Also, please be sure to have read all of the README. It contains a lot of information regarding known system and user issues.

• Enable moveonenospc option
• Share mergerfs volume with Samba
• Mount Samba share on a different system (OS X 10.12.6)
• Use DaVinci Resolve's clone function to copy a file that is larger than the remaining space on one of the merged drives (my current problem involves copying a 371GB file to a drive with ~150GB free space). A lfs create policy makes this easier to reproduce.
• Allow the drive to fill up & moveonenospc to trigger, so that the file gets copied to a drive with more space.
• While file is being moved, use Finder on the client system to access the Samba share, select a file that is hosted on the drive that is the source for moveonenospc. Access the file by pressing space to use Finder's "Preview" function. Alternately, Finder trying to generate a thumbnail can also trigger this.
• Finder will freeze for multiple minutes as it tries to access the file system.

I recognize that accessing the merged filesystem through Samba is likely an exacerbating factor here, but the problem is likely to exist without it.

Perhaps a solution is to check free space before a file is created (thus avoiding the moveonenospc in the first place)? In my specific example, running "ls -lh" on the underlying drive showed that a 371GB file had been allocated to the drive, but the directory only contained 150GB with zero free space. Simultaneously, the destination drive also showed a 371GB file had been allocated. So, apparently the filesystem knew that the file size needed to be larger than the remaining space.

Eventually (after about an hour), Davinci Resolve decided the copy had timed out, and it cancelled the file transfer, and both allocated files disappeared.

System information

Please provide as much of the following information as possible:

• [ 2.29 ] mergerfs version: mergerfs -V
• [ /media/SB*/Storybubble /media/Storybubble\040Main fuse.mergerfs defaults,allow_other,fsname=Storybubble\040Main,dropcacheonclose=true,moveonenospc=true,ignorepponrename=true,minfreespace=20G,category.create=lfs,use_ino,x-systemd.requires=/media/SB-Archive1,x-systemd.requires=/media/SB-Archive2,x-systemd.requires=/media/SB-Archive3,x-systemd.requires=/media/SB-Archive4,x-systemd.requires=/media/SB-Archive5,x-systemd.requires=/media/SB-Archive6,x-systemd.requires=/media/SB-Archive7,x-systemd.requires=/media/SB-Archive8,x-systemd.requires=/media/SB-TIEB ] mergerfs settings: cat /etc/fstab or the command line arguments
• [ Linux storybubble-Storage 4.15.0-115-generic #116-Ubuntu SMP Wed Aug 26 14:04:49 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux ] Linux version: uname -a
• [ smbd Version 4.7.6-Ubuntu, OS X 10.12.6, Davinci Resolve 15.3.1.003 ] Versions of any additional software being used
• [ <loopback devices for snap & boot drive omitted> /dev/sdl1 7.3T 7.3T 14G 100% /media/parity1 /dev/sda1 7.3T 7.3T 7.4G 100% /media/SB-Archive6 /dev/sdj1 7.3T 7.2T 150G 98% /media/SB-TIEB /dev/sdd1 7.3T 7.3T 13G 100% /media/parity3 /dev/sdk1 7.3T 7.3T 20G 100% /media/SB-Archive5 /dev/sdc1 7.3T 7.3T 13G 100% /media/parity2 /dev/sdg1 3.7T 3.7T 19G 100% /media/SB-Archive8 /dev/sde1 3.7T 3.7T 16G 100% /media/SB-Archive7 /dev/sdi1 7.3T 2.8T 4.6T 38% /media/SB-Archive2 /dev/sdf1 7.3T 7.3T 15G 100% /media/SB-Archive1 /dev/sdm1 7.3T 7.3T 19G 100% /media/SB-Archive3 /dev/sdh1 3.7T 3.7T 19G 100% /media/SB-Archive4 Storybubble Main 55T 50T 4.9T 92% /media/Storybubble Main] List of drives, filesystems, & sizes: df -h
• [ app.strace.txt (one of many smbd instances running at the time; I chose the most recent one) ] strace of application having problem: strace -f -o /tmp/app.strace.txt <cmd> or strace -f -p <appPID> -o /tmp/app.strace.txt
• [ This generated an 888MB file before the copy failed, and appeared to include some of my raw data. Omitted for privacy and practicality reasons ] strace of mergerfs while app tried to do it's thing: strace -f -p <mergerfsPID> -o /tmp/mergerfs.strace.txt

[trapexit]

Please use the most recent version and test with a simplified setup. I can't control what macOS or Samba do and they in the past have had poor behaviors.

Perhaps a solution is to check free space before a file is created (thus avoiding the moveonenospc in the first place)?

That's impossible. The creation of a file has zero information on how the file will be used or how much data is incoming. The moving of a file does check if the target drive has space before it moves the file the "moveonenospc" behavior is triggered.

mergerfs simply copies the file inside the write call. There is no other way to do it. And if it has to copy a ton of data that's what it has to do. Finder might simply be not handling the blocking well and there is nothing I can do about that if so. I don't have a modern Mac so I can't test this but regular moveonenospc usage seems fine on the host itself.

[Devonavar]

I was afraid that would be the answer. I'm not willing to experiment too much with a production machine right now, including changing versions. I can't afford to risk days of downtime for a rebuild. I was hoping it might be something simple like changing CPU priority, but I can imagine it being quite complicated otherwise!

So, for now, I'll put up with it and you can close this issue I guess. I'll try and do further testing at some point in the future when I'm not so dependent on my machine. Please keep an eye on moveonenospc processing times!

[trapexit]

We can leave it open but without confirming whose having the issue I can't really do much. The process itself is no more or less heavy than a normal file copy. The only difference from the client side is that it blocks longer than normal as the transfer happens. And that's not something I'm able to do anything about practically speaking.