NFS & git issue

[trapexit]

(Sorry to resurrect an old thread. I can open a new one, if needed.)

I'm still having this issue. I cannot push local git changes to an NFS-over-mergerfs remote.

mergerfs version: 2.32.2 nfs version: 4.2 (nfs-kernel-server: Installed: 1:1.3.4-2.5ubuntu3.3)

On the server:

#/etc/fstab
/mnt/disk*      /mergershare    fuse.mergerfs   defaults,allow_other,cache.files=off,use_ino,category.create=msplfs,ignorepponrename=true,moveonenospc=mspmfs,minfreespace=50G,nfsopenhack=git,fsname=mergerfsPool      0       0

#/etc/exports
/mergershare/srv/git    client(rw,sync,no_subtree_check,fsid=13)
/mnt/disk1/srv/git client(rw,sync,no_subtree_check,fsid=14)

On the client:

#/etc/fstab
server:/mergershare/srv/git    /srv/git  nfs defaults,noatime,_netdev    0   0
server:/mnt/disk1/srv/git    /tmp/rgit  nfs defaults,noatime,_netdev    0   0

This is what happens:

/tmp$ git clone /srv/git/test.git/

...

/tmp/test$ git push
Counting objects: 3, done.
Writing objects: 100% (3/3), 232 bytes | 232.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0)
remote: error: insufficient permission for adding an object to repository database /srv/git/test.git/./objects/incoming-rtckJd
remote: fatal: failed to write object
error: remote unpack failed: unpack-objects abnormal exit
To /srv/git/test.git/
 ! [remote rejected] master -> master (unpacker error)
error: failed to push some refs to '/srv/git/test.git/'

Here is a test with NFS without the mergerfs layer

/tmp$ git clone /tmp/rgit/test.git/ test2

...

/tmp/test2$ git push
Counting objects: 3, done.
Writing objects: 100% (3/3), 235 bytes | 235.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0)
To /tmp/rgit/test.git/
   e2ac369..9510d07  master -> master

Originally posted by @dpantel in https://github.com/trapexit/mergerfs/issues/626#issuecomment-820739491

[trapexit]

@dpantel Please provide more information as described in the docs. OS? Version of everything involved? Etc.

[dpantel]

Here is the extra info:

$ uname -a #client
Linux client 4.15.0-141-generic #145-Ubuntu SMP Wed Mar 24 18:08:07 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

$ uname -a #server
Linux server 5.4.0-71-generic #79-Ubuntu SMP Wed Mar 24 10:56:57 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

$ df -h
/dev/mapper/disk1        916G  822G   49G  95% /mnt/disk1
/dev/mapper/disk2        3.6T  1.2T  2.3T  34% /mnt/disk2
mergerfsPool             4.5T  2.0T  2.4T  46% /mergershare

$ lsblk
sdd               8:48   0   3.7T  0 disk  
└─disk2         253:6    0   3.7T  0 crypt /mnt/disk2
sde               8:64   0 931.5G  0 disk  
└─disk1         253:5    0 931.5G  0 crypt /mnt/disk1

fuse version 2.9.9-3

git version 2.17.1 (client) git version 2.25.1 (server)

app.strace.txt mergerfs.strace.txt

[trapexit]

https://github.com/trapexit/mergerfs#nfsopenhack

if set to git it will only perform the hack when the path in question includes /.git/. all will result it it applying anytime a readonly file which is empty is opened for writing.

14653 16:57:19.893397 openat(AT_FDCWD, "dp/srv/git/test.git/./objects/incoming-FevbPK/00/tmp_obj_ix4TEc", O_RDWR|O_CREAT|O_EXCL, 0444) = -1 EACCES (Permission denied) <0.002797>

It doesn't have '/.git/' in the path so as configured it didn't apply the hack. Did you try turning on all?

[dpantel]

I’ve tried changing that setting to “all” and remounting with nonempty option and got the same result. Is there an xattr command to change that setting, just to be sure it actually got changed.

Also, I understand that this whole thing is a hack for some abnormal behaviors outside of your control, and searching for ‘.git/‘ is easy, but consider that a bare repo won’t have that directory (as in my case). I wonder if there is another way to identify a git repo.

Thanks for your help.

[trapexit]

xattr -p user.mergerfs.nfsopenhack /media/.mergerfs

Of course there are other ways but you have to consider the costs of performing the checks. What would you suggest?

https://github.com/trapexit/mergerfs/blob/master/src/fuse_open.cpp#L77

That's the code. As you can see I try to limit the cost of such checks because if enabled they have to be evaluated every single time a file is opened and depending on the workload can drastically increase costs. FUSE + network filesystem + whatnot is already pretty high relative overheads.

[trapexit]

If all is enabled and you're sure it's not working then please provide straces again because the mergerfs strace provided before isn't useful since it doesn't actually perform the hack.

[dpantel]

Ok, I changed nfsopenhack to "all" and it works fine now. Having seen the code, I get why the hack wasn't being triggered with the "git" setting in my case.

I did a little dive into git's source for git rev-parse --is-inside-git-dir to see if there is a simple test that git uses to identify its own directories, but it's kind of a mess that requires setting up the git environment to work.

The only thought I have for tweaking the hack is maybe stripping the leading slash from fullpath_.find("/.git/"), since by convention, bare repository directories are named "<repo-name>.git".