NFSv4 Permission denied even with correct permissions

[tailtwo]

Issue Report

Bug

Some files hosted on a NFSv4 export with correct permissions randomly cannot be read with any user (Permission denied). A ls -l (emphasis on th -l) solve the problem. Maybe a getattr command is "fixing" it?

Container Linux Version

$ cat /etc/os-release
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=2023.5.0
VERSION_ID=2023.5.0
BUILD_ID=2019-03-09-0138
PRETTY_NAME="Container Linux by CoreOS 2023.5.0 (Rhyolite)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"

Environment

Container Linux on bare metal and ignition config.

Expected Behavior

Reading a file with correct permissions on a NFS mount returns the file content without problems.

Actual Behavior

Reading a file returns Permission denied with any account (even root) even when the correct permissions.

Reproduction Steps

1. Mount a NFS export on the host with the following options (Kubernetes default) : rw,relatime,vers=4.2,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.20.6,local_lock=none,addr=192.168.20.4
2. Write a file in the mounted export
3. Issue a command that read the file, sometimes and randomly the file maybe unreadable because of a permission problem.

Other Information

This behaviour appeared on 2023.5.0, the previous version (1967.6.0) worked for months without this issue. I suspect a kernel issue. I attached a tcpdump capture during a cat on a faulty file (sorry for the amount of packets but I couldn't narrow my scope more).

capture.zip

EDIT: Here is a strace of a cat command on a faulty file :

execve("/usr/bin/cat", ["cat", "vol-01/chap-06/d0eac605-08c0-43f"...], [/* 26 vars */]) = 0
brk(0)                                  = 0x563ba621f000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=31149, ...}) = 0
mmap(NULL, 31149, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f1888404000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240\20\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1869216, ...}) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1888402000
mmap(NULL, 3976832, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f1887e1d000
mprotect(0x7f1887fde000, 2097152, PROT_NONE) = 0
mmap(0x7f18881de000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c1000) = 0x7f18881de000
mmap(0x7f18881e4000, 16000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f18881e4000
close(3)                                = 0
arch_prctl(ARCH_SET_FS, 0x7f1888403500) = 0
mprotect(0x7f18881de000, 16384, PROT_READ) = 0
mprotect(0x563ba4a3e000, 4096, PROT_READ) = 0
mprotect(0x7f188840c000, 4096, PROT_READ) = 0
munmap(0x7f1888404000, 31149)           = 0
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 2), ...}) = 0
openat(AT_FDCWD, "vol-01/chap-06/d0eac605-08c0-43fd-9f3e-0e3c90e8de24.properties", O_RDONLY) = -1 EACCES (Permission denied)
write(2, "cat: ", 5cat: )                    = 5
write(2, "vol-01/chap-06/d0eac605-08c0-43f"..., 62vol-01/chap-06/d0eac605-08c0-43fd-9f3e-0e3c90e8de24.properties) = 62
write(2, ": Permission denied", 19: Permission denied)     = 19
write(2, "\n", 1
)                       = 1
close(1)                                = 0
close(2)                                = 0
exit_group(1)                           = ?
+++ exited with 1 +++

[ajeddeloh]

I agree this sounds like a kernel issue. Can you reproduce on the latest alpha or beta? They're running slightly newer kernels which will tell us if the fix is already upstream.

[tailtwo]

Mounted the NFS export on manjaro with kernel 5.0.5-1-MANJARO and the problematic files are perfectly readable without any workaround. Will try CoreOS beta branch next week.

[ajeddeloh]

It's worth noting the CL will remain on the 4.19.x stable kernel branch through it's end of life. It's useful to know that it works with a 5.0.x kernel, but we won't be shipping a 5.0.x kernel.