mergerfs
mergerfs copied to clipboard
trapexit
Users lose access to filesystem after 2.33.?
Following an upgrade from mergerfs-2.32.5-1.el7.x86_64 -> mergerfs-2.33.3-1.el7.x86_64 on Centos 7.9 users could only see the contents of the mergerfs mount point. All directories at this point appeared empty.
For root: [root@Nethserver ~]# ls -l /MyPool/ total 636608 drwxrwx--- 5 root [email protected] 4096 Jan 25 21:17 Backup -rw------- 1 root root 651839138 Jan 27 05:02 content -rw------- 1 root root 0 Jan 27 16:14 content.lock drwxrws--- 5 root [email protected] 197 Jan 26 16:00 DVR drwxrws--- 2 root [email protected] 6 Apr 8 2021 Games drwxrwx--- 4 root [email protected] 50 Sep 21 10:57 iTunes drwxrws--- 2 root [email protected] 6 Aug 20 2019 Lotus Stuff drwxrws--- 2 root [email protected] 8192 Dec 3 14:33 Movies drwxrws--- 19 root [email protected] 4096 Oct 5 2020 Original Movies drwxrws---+ 2 root [email protected] 6 Mar 15 2015 Swapsies drwxrwx--- 16 root [email protected] 4096 Nov 19 16:53 TV Shows [root@Nethserver ~]# ls -l /MyPool/iTunes/ total 24 drwxrws---+ 2 root [email protected] 6 Dec 7 2016 Automatically Add to iTunes -rw-rw---- 1 root [email protected] 4953 Nov 26 18:41 cd3o.m3u drwxr-sr-x 2 root [email protected] 6 Nov 21 2019 Downloads drwxrws--- 3 root [email protected] 28 Jun 27 2019 LossLess drwxrws---+ 2 root [email protected] 87 Dec 7 2016 Mobile Applications drwxrwx---+ 11 root [email protected] 4096 Oct 13 14:38 Music drwxrws---+ 5 root [email protected] 90 Nov 21 2019 Podcasts drwxrwx--- 2 root [email protected] 41 Sep 21 10:57 Voice Memos [root@Nethserver ~]#
But for a user: [[email protected]@Nethserver ~]$ ls -l /MyPool/ total 636608 drwxrwx--- 5 root [email protected] 4096 Jan 25 21:17 Backup -rw------- 1 root root 651839138 Jan 27 05:02 content -rw------- 1 root root 0 Jan 27 16:14 content.lock drwxrws--- 5 root [email protected] 197 Jan 26 16:00 DVR drwxrws--- 2 root [email protected] 6 Apr 8 2021 Games drwxrwx--- 4 root [email protected] 50 Sep 21 10:57 iTunes drwxrws--- 2 root [email protected] 6 Aug 20 2019 Lotus Stuff drwxrws--- 2 root [email protected] 8192 Dec 3 14:33 Movies drwxrws--- 19 root [email protected] 4096 Oct 5 2020 Original Movies drwxrws---+ 2 root [email protected] 6 Mar 15 2015 Swapsies drwxrwx--- 16 root [email protected] 4096 Nov 19 16:53 TV Shows [[email protected]@Nethserver ~]$ ls -l /MyPool/iTunes/ total 0 [[email protected]@Nethserver ~]$
No other changes other than the mergerfs upgrade took place. Erasing the 2.33.3 version and reinstalling 2.32.5 resolved the issue.
The mount command used: /poolmount/Data? /MyPool fuse.mergerfs defaults,allow_other,direct_io,use_ino,minfreespace=50G,func.getattr=newest,category.create=mfs,fsname=mergerfs 0 0
As an aside, the update from 2.32.? -> 2.33.? also throws a conflict error:
[root@Nethserver ~]# yum update mergerfs-2.33.1-1.el7.x86_64.rpm Loaded plugins: changelog, elrepo, fastestmirror, nethserver_events Examining mergerfs-2.33.1-1.el7.x86_64.rpm: mergerfs-2.33.1-1.el7.x86_64 Marking mergerfs-2.33.1-1.el7.x86_64.rpm as an update to mergerfs-2.32.5-1.el7.x86_64 Resolving Dependencies --> Running transaction check ---> Package mergerfs.x86_64 0:2.32.5-1.el7 will be updated ---> Package mergerfs.x86_64 0:2.33.1-1.el7 will be an update --> Finished Dependency Resolution --> Finding unneeded leftover dependencies Found and removing 0 unneeded dependencies
Dependencies Resolved
================================================================================ Package Arch Version Repository Size
Updating: mergerfs x86_64 2.33.1-1.el7 /mergerfs-2.33.1-1.el7.x86_64 617 k
Transaction Summary
Upgrade 1 Package
Total size: 617 k Is this ok [y/d/N]: y Downloading packages: Running transaction check Running transaction test
Transaction check error: file /usr/share/man/man1/mergerfs.1.gz from install of mergerfs-2.33.1-1.el7.x86_64 conflicts with file from package mergerfs-2.32.5-1.el7.x86_64
Error Summary
[root@Nethserver ~]#
Cheers.
Can you provide the details requested in the ticket template? I really can't help debug issues if you don't provide a minimal reproducible example and / or the details requested.
System information:
[[email protected]@Nethserver ~]$ uname -a Linux Nethserver.BogoLinux.net 3.10.0-1160.53.1.el7.x86_64 #1 SMP Fri Jan 14 13:59:45 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux [[email protected]@Nethserver ~]$ mergerfs -V mergerfs version: 2.33.3 [[email protected]@Nethserver ~]$ [[email protected]@Nethserver ~]$ df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 7.8G 0 7.8G 0% /dev tmpfs 7.8G 4.0K 7.8G 1% /dev/shm tmpfs 7.8G 13M 7.8G 1% /run tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup /dev/mapper/cl-root 32G 8.4G 24G 27% / /dev/sdb 3.7T 1.6T 2.1T 44% /poolmount/Parity1 /dev/sdh 1.9T 1.4T 520G 73% /poolmount/Data1 /dev/sdj 1.9T 1.1T 791G 58% /poolmount/Data6 /dev/sda 3.7T 2.9T 762G 80% /poolmount/DataB /dev/sdl 1.9T 969G 895G 52% /poolmount/Data2 /dev/sdd 1.9T 942G 921G 51% /poolmount/Data8 /dev/sdc 1.9T 950G 913G 51% /poolmount/Data9 /dev/sdg 1.9T 1.4T 514G 73% /poolmount/Data3 /dev/sdk 3.7T 1.9T 1.8T 53% /poolmount/Data4 /dev/sdf 1.9T 926G 937G 50% /poolmount/Data5 /dev/sde 1.9T 648G 1.2T 35% /poolmount/DataA /dev/sdi 1.9T 1.2T 660G 65% /poolmount/Data7 /dev/sdm1 1014M 247M 768M 25% /boot /dev/mapper/cl-PlexLib 20G 4.9G 16G 25% /var/lib/plexmediaserver tmpfs 1.6G 0 1.6G 0% /run/user/1568801106 mergerfs 24T 14T 9.7T 60% /MyPool [[email protected]@Nethserver ~]$ lsblk -f NAME FSTYPE LABEL UUID MOUNTPOINT sda xfs DataB 2c3641a0-9ec5-470d-bd9d-27d2e54d4fd6 /poolmount/DataB sdb xfs Parity1 c4dfcb52-a2fe-4571-8274-8638aff36cca /poolmount/Parity sdc xfs Data9 9c3c0609-2c09-47c5-915d-8e8520a2301b /poolmount/Data9 sdd xfs Data8 f225a749-93eb-447b-9e3d-aed1d5dc33da /poolmount/Data8 sde xfs DataA b0ed33c5-7e34-4a91-b05f-0bf5041cda33 /poolmount/DataA sdf xfs Data5 4b4b4377-e98b-4fd2-b2f7-948829308f38 /poolmount/Data5 sdg xfs Data3 76a707b5-89fe-488c-9721-5285381b17d4 /poolmount/Data3 sdh xfs Data1 9f12fc75-405f-439e-8a68-f08bb6402341 /poolmount/Data1 sdi xfs Data7 41e428c3-1b67-4e9f-a9d9-71b981f4b59f /poolmount/Data7 sdj xfs Data6 5ef6630d-b91c-40c2-b423-cb38fd8d91c4 /poolmount/Data6 sdk xfs Data4 2f01e9a8-5222-4f24-8420-983c4fae5453 /poolmount/Data4 sdl xfs Data2 9bbe4400-d1b0-418d-baf3-72970e2b91c2 /poolmount/Data2 sdm ├─sdm1 xfs 24f31a2d-cecb-4238-aa6b-00ed87db1db4 /boot └─sdm2 LVM2_mem PTVeke-5Gpq-kV0N-7ud3-omlj-eZJn-EKZZjd ├─cl-root xfs 55901af1-c334-4fe7-9210-a6584e4b9b32 / ├─cl-swap swap 0f4d5c66-91d9-4d95-af43-f8e65ac6addd [SWAP] └─cl-PlexLib xfs PlexLib 38959cdf-bc70-40a3-8580-f388764b3657 /var/lib/plexmedi [[email protected]@Nethserver ~]$
The mergerfs mount is in the initial report.
For me, to reproduce is simply upgrade from version 2.32.5 to any 2.33 version. (By removing 2.32.5 and installing 2.33.? because of the merge conflict mentioned above). To "fix" the issue, I just revert back to 2.32.5.
Not sure what else I can give that might be useful.
The docs explain what is useful:
https://github.com/trapexit/mergerfs/blob/master/.github/ISSUE_TEMPLATE/bug_report.md https://github.com/trapexit/mergerfs#support
A completely stripped down version of the situation. Does it happen with only one branch? or is it only multiple branches? Do other commands work or is it just readdir that fails? Have you taken straces? Is it different with different permissions.
Many people, including myself, use the latest versions without issue so I need the most simplest version of the issue to attempt to reproduce. I don't have a CentOS system. I don't have your setup. I don't have those filesystems with those files and permissions. I need something as simple as possible that replicates the issue and all the relevant data to debug without having access to your machine. That includes all the details I've requested in the docs.
Still working on setting up a minimal environment, but in the meantime, hoping it helps, strace's of a simple "ls" command on both my working system and the failing one. Files are actually tgz, but can't upload that extension.
If this is still an issue with the latest release please reopen. I was never able to reproduce even on a centos 7 system.