Slow writes in MergerFS

[EverettPorter]

trafficstars

Describe the bug

Writes via MergerFS are very much slower than writing directly to the same storage. This has been true over many versions, including the latest. Reads are somewhat slower than directly from underlying filesystems, but this difference is much less pronounced. I have tried a few combinations of caching options but have not yet found anything that helps significantly. The current settings below are from today, including the 4 threads and cache settings, are from today's tests. Before then they were basically the default settings. I'm fully aware that this issue may be due to my ignorance of the underlying systems! Any help would be greatly appreciated.

To Reproduce

bonnie++ -u xx -f (on MergerFS volume) Version 1.97 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP newserver.poly 62G 166016 27 141270 25 1357946 70 557.0 18 Latency 3336us 538ms 437ms 519ms Version 1.97 ------Sequential Create------ --------Random Create-------- newserver.poly -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 6185 13 +++++ +++ 9279 13 5852 12 +++++ +++ 8632 13 Latency 614us 136us 10314us 6940us 118us 22616us 1.97,1.97,newserver.poly,1,1639744291,62G,,,,166016,27,141270,25,,,1357946,70,557.0,18,16,,,,,6185,13,+++++,+++,9279,13,5852,12,+++++,+++,8632,13,,3336us,538ms,,437ms,519ms,614us,136us,10314us,6940us,118us,22616us

same, but on underlying filesystem (same pool member as used for above). Version 1.97 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP newserver.poly 62G 1072775 98 605592 66 1893393 87 543.8 47 Latency 75us 573ms 110ms 473ms Version 1.97 ------Sequential Create------ --------Random Create-------- newserver.poly -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ Latency 8791us 184us 19304us 8558us 73us 3304us 1.97,1.97,newserver.poly,1,1639743840,62G,,,,1072775,98,605592,66,,,1893393,87,543.8,47,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,,75us,573ms,,110ms,473ms,8791us,184us,19304us,8558us,73us,3304us

Please simplify the reproduction as much as possible.

• Unless it is dependenat on multiple branches, use a single branch
• Use standard tooling if possible (touch,truncate,rm,rmdir,ln,etc.)

Expected behavior

Somewhat slower in MergerFS, but not such a dramatic difference.

System information:

• OS, kernel version: uname -a
• Linux newserver.poly 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• mergerfs version: mergerfs -V mergerfs version: 2.32.6 (also tried with latest).
• mergerfs settings
• /mnt/WD18Tx19-2021-9_PC/PolyCast2:/mnt/HGST14TBx18-2/PolyCast4/PolyCast4:/mnt/WD18TBx18-1/PolyCast1-2:/mnt/HGST14TBx18/PolyCast3:/mnt/WD18Tx16-2021-9/PolyCast5:/mnt/WD18Tx16-2021-9/PolyCast6:/mnt/lsi_ext5.1/PolyCast7:/mnt/SG12TB/PC8 /mnt/PC_Merge fuse.mergerfs defaults,allow_other,ignorepponrename=true,use_ino,hard_remove,minfreespace=500G,category.create=mfs,category.search=all,fsname=PC_Merge,cache.files=auto-full,cache.writeback=true,dropcacheonclose=true,splice_write,threads=4,statfs=base 0 0
• List of drives, filesystems, & sizes:
  • df -h
  • /dev/sda5 25G 18G 7.1G 72% / /dev/sda1 1014M 243M 772M 24% /boot /dev/sda2 25G 701M 25G 3% /home /dev/sdb 246T 216T 31T 88% /mnt/WD18TBx18-1 /dev/sdh 230T 214T 16T 94% /mnt/WD18Tx16-2021-9 /dev/sdf 191T 169T 23T 89% /mnt/HGST14TBx18-2 /dev/sdg 153T 144T 9.1T 95% /mnt/SG12TB /dev/sde1 62T 62T 517G 100% /mnt/lsi_ext5.1 /dev/sde2 62T 59T 3.6T 95% /mnt/lsi_ext5.2 /dev/sdd 279T 123T 157T 44% /mnt/WD18Tx19-2021-9_PC /dev/sdc 191T 179T 13T 94% /mnt/HGST14TBx18 //192.168.9.1/ftp 848G 313G 536G 37% /mnt/ftp tmpfs 3.1G 0 3.1G 0% /run/user/506 PC_Merge 1.1P 872T 181T 83% /mnt/PC_Merge mergerfsPool 791T 573T 219T 73% /mnt/sambashare/SessMed
  • lsblksda 8:0 0 223.1G 0 disk ├─sda1 8:1 0 1G 0 part /boot ├─sda2 8:2 0 25G 0 part /home ├─sda3 8:3 0 15.6G 0 part [SWAP] ├─sda4 8:4 0 1K 0 part ├─sda5 8:5 0 25G 0 part / ├─sda6 8:6 0 1G 0 part ├─sda7 8:7 0 25G 0 part ├─sda8 8:8 0 15.6G 0 part [SWAP] └─sda9 8:9 0 25G 0 part sdb 8:16 0 245.6T 0 disk /mnt/WD18TBx18-1 sdc 8:32 0 191T 0 disk /mnt/HGST14TBx18 sdd 8:48 0 278.3T 0 disk /mnt/WD18Tx19-2021-9_PC sde 8:64 0 123.7T 0 disk ├─sde1 8:65 0 61.9T 0 part /mnt/lsi_ext5.1 └─sde2 8:66 0 61.9T 0 part /mnt/lsi_ext5.2 sdf 8:80 0 191T 0 disk /mnt/HGST14TBx18-2 sdg 8:96 0 152.8T 0 disk /mnt/SG12TB sdh 8:112 0 229.2T 0 disk /mnt/WD18Tx16-2021-9
• A strace of the application having a problem:
  • strace -fvTtt -s 256 -o /tmp/app.strace.txt <cmd>
  • strace -fvTtt -s 256 -o /tmp/app.strace.txt -p <appPID>
• strace of mergerfs while app tried to do it's thing:
  • strace -fvTtt -s 256 -p <mergerfsPID> -o /tmp/mergerfs.strace.txt

Additional context

Add any other context about the problem here.

[trapexit]

Have you read through the documentation on the topic of performance?