Unable to git clone on top of NFS share with Ganesha v6.

[ralcolea]

I started having issues with one of the SaunaFS integration tests for Ganesha after updating our current SaunaFS FSAL (Compatible with Ganesha v4.3) with the version of SaunaFS FSAL released at the Ganesha repository (Compatible with Ganesha v6).

The failing test is the cthon test, that runs basically in two steps:

1. Clone and build cthon repository on top of the NFS share exported with Ganesha: FAILED
2. Run cthon tests for NFS share: SUCCESS

Previously, this test passed successfully. However, after upgrading, the git clone command fails with permission denied. If I run the git clone command on top of the SaunaFS mount, then the git clone works as expected and the test passes successfully.

I'm using this configuration file for the test:

NFSV4 {
	Grace_Period = 5;
}
EXPORT
{
	Attr_Expiration_Time = 0;
	Export_Id = 99;
	Path = /data;
	Pseudo = /data;
	Access_Type = RW;
	FSAL {
		Name = SaunaFS;
		hostname = localhost;
		port = ${saunafs_info_[matocl]};
		# How often to retry to connect
		io_retries = 5;
		cache_expiration_time_ms = 2500;
	}
	Protocols = 4;
	CLIENT {
		Clients = localhost;
	}
}
LOG {
	COMPONENTS {
		NFSPROTO=FULL_DEBUG;
		NFS_V4=FULL_DEBUG;
		FSAL=FULL_DEBUG;
		EXPORT=FULL_DEBUG;
		NFS_V4_ACL=FULL_DEBUG;
	}
}

The most important steps in the test are below:

mkdir -p ${TEMP_DIR}/mnt/ganesha
mkdir -p ${info[mount0]}/data

sudo /usr/bin/ganesha.nfsd -f ${info[mount0]}/ganesha.conf -L /tmp/cthon.log
assert_eventually 'showmount -e localhost'

sudo mount -vvvv localhost:/data $TEMP_DIR/mnt/ganesha

# Run connectathon nfs suite
cd $TEMP_DIR/mnt/ganesha

mkdir cthon_tests
export NFSTESTDIR=$TEMP_DIR/mnt/ganesha/cthon_tests

git clone https://github.com/leil-io/cthon04.git # This is the failing step
cd cthon04
make all

./runtests -b -n
./runtests -l -n
./runtests -s -n

The log produced during the test is the following: cthon.log

Did somebody have this issue? Any ideas about how to solve it? Thanks in advance,

[xiaods]

you need ask SaunaFS community.

[ralcolea]

Thank you for your answer @xiaods.

I'm one of the maintainers of the SaunaFS community and as I said in my first comment, git clone works well with Ganesha v4.3, the version we currently support. However, after migrating our SaunaFS FSAL to v6, git clone stopped working due to permission issues. We didn't do changes in our FSAL, that's why I'm a little confused about what could be the cause of this issue.

[ffilz]

A working (4.3) and a non-working (6) tcp trace would help.

I can't think of any permission processing changes, so not sure what it might be.

Are you using the latest, V6-dev.14?

[ralcolea]

Hi @ffilz,

Thank you for your answer.

At the moment of creating the issue, V6-dev.11 was the latest version available. However, I updated to V6-dev.14 and I have the same issue.

Below I share tcp traces generated with tcpdump for working (V4.3) and non-working (V6-dev.14) versions. Please, confirm whether this is what you need or if you need some other logs/traces. ganesha-v6-dev.14 ganesha-v6-dev.14-issue-1132.zip ganesha-v6-dev.14-issue-1132.log

ganesha-v4.3 ganesha-v4.3-issue-1132.zip ganesha-v4.3-issue-1132.log

Thank you in advance,

[ffilz]

Hmm, the traces have no NFS traffic in them...

[ralcolea]

Hi @ffilz,

Sorry for being absent during the last days.

I updated TCP traces to include NFS traces and also added the log generated when running the tests with each Ganesha version. I modified my previous comment to update the new information.

Please, let me know if the new information is correct.

Thanks for your support,

[derekbit]

Running into the issue while using v6.5. It is not observed in v5.8 and v5.9. @ralcolea Does the issue still remain on your side?

[derekbit]

Hello @ffilz We are experiencing the same issue in nfs-ganesha ~~v6.4~~ v6.5. More information is in https://github.com/longhorn/longhorn/issues/10621 and the nfs server log is perm-denied.txt. Could you help check the issue? Thank you.

root@rancher60-master:/home/vagrant# k exec -it  test-volume-mount /bin/sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
/git # adduser --uid 2002 test3
Changing password for test3
New password:
Bad password: too weak
Retype password:
passwd: password for test3 changed by root
/git # cd /mnt/test; mkdir test; chown test3:test3 test
/mnt/test # cd test/
/mnt/test/test # su test3
/mnt/test/test $ git clone https://github.com/longhorn/longhorn.git
Cloning into 'longhorn'...
remote: Enumerating objects: 11976, done.
remote: Counting objects: 100% (2228/2228), done.
remote: Compressing objects: 100% (164/164), done.
remote: Total 11976 (delta 2138), reused 2066 (delta 2064), pack-reused 9748 (from 3)
Receiving objects: 100% (11976/11976), 16.17 MiB | 21.78 MiB/s, done.
fatal: could not open '/mnt/test/test/longhorn/.git/objects/pack/tmp_pack_KgwS2T' for reading: Permission denied
fatal: fetch-pack: invalid index-pack output
/mnt/test/test $ command terminated with exit code 137

[ralcolea]

Running into the issue while using v6.5. It is not observed in v5.8 and v5.9. @ralcolea Does the issue still remain on your side?

I assume we still do. We have not changed anything in our FSAL since my last answer. I shared updated logs, but we did not receive new feedback.

[jswoodward]

I believe this is the same thing I saw which is fixed in 6.5 by #1219

[derekbit]

I believe this is the same thing I saw which is fixed in 6.5 by #1219

@jswoodward Thank you. Sorry for the wrong version in my previous comment. Unfortunately, the issue remains in nfs-ganesha v6.5.

[ffilz]

Can you break out from the traces the sequence of ops that leads to the permission error?

I'm sorry, I'm absolutely swamped and can't dive into every issue in detail.

[ffilz]

I believe this is the same thing I saw which is fixed in 6.5 by #1219

@jswoodward Thank you. Sorry for the wrong version in my previous comment. Unfortunately, the issue remains in nfs-ganesha v6.5.

Ah yes, if that's the case, then we need a backport to V6-stable.

Could you try SaunaFS on V7-dev.11?

We need to think about a V6.6 merge with a bunch of backports.

[jswoodward]

I believe this is the same thing I saw which is fixed in 6.5 by #1219

@jswoodward Thank you. Sorry for the wrong version in my previous comment. Unfortunately, the issue remains in nfs-ganesha v6.5.

Ah yes, if that's the case, then we need a backport to V6-stable.

Could you try SaunaFS on V7-dev.11?

We need to think about a V6.6 merge with a bunch of backports.

Apologies... I completely missed it didn't make it in time for 6.5.

[ralcolea]

Hi @ffilz and @jswoodward,

I tested SaunaFS FSAL on V7-dev.11 and the issue persists. I also applied the patch created by @jswoodward to solve issue #1219, but the issue is still present.

One difference I noticed compared to issue #1219 is that in my case, the failure occurs during the creation of the file, whereas in #1219 it was during the open operation.

I'm not sure whether these operations are handled together or separately within Ganesha.

Additionally, I want to highlight that the git clone operation is being run by a non-privileged user —the same user that runs our test suite.