Commencing io-test on an NFS share raises permission denied error

[rustyscottweber]

This will fail unless the server has the share created using the no_root_squash option in the exports file.

Environment: Ubuntu 20.04

fio version: 3.27

Reproduction steps Process to recreate:

1. Create an NFS share.
2. Export NFS share to system where fio will run from.
3. Create fio test files, using fio, on nfs share.
4. Commence fio test using newly created files on the NFS share

root@r11i1:~# fio --aux-path /tmp/fio_AVIiUF /tmp/fio_AVIiUF/fio_job_gxbgOt.ini r11i1_job0_zZRgcjyC.IO_TEST: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16 r11i1_job1_wmbEEDWF.IO_TEST: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16 r11i1_job0_KOyqHgFS.IO_TEST: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16 r11i1_job1_TPbisOqo.IO_TEST: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16 fio-3.27 Starting 4 processes fio: pid=10424, err=13/file:filesetup.c:805, func=open(/tmp/mount_192_168_201_1_universal_share/r11i1_job0_KOyqHgFS.IO_TEST), error=Permission denied fio: pid=10423, err=13/file:filesetup.c:805, func=open(/tmp/mount_192_168_200_1_universal_share/r11i1_job1_wmbEEDWF.IO_TEST), error=Permission denied fio: pid=10425, err=13/file:filesetup.c:805, func=open(/tmp/mount_192_168_201_1_universal_share/r11i1_job1_TPbisOqo.IO_TEST), error=Permission denied fio: pid=10422, err=13/file:filesetup.c:805, func=open(/tmp/mount_192_168_200_1_universal_share/r11i1_job0_zZRgcjyC.IO_TEST), error=Permission denied

~# cat /tmp/fio_AVIiUF/fio_job_gxbgOt.ini [global] direct=1 [r11i1_job0_zZRgcjyC.IO_TEST] filename=/tmp/mount_192_168_200_1_universal_share/r11i1_job0_zZRgcjyC.IO_TEST rwmixread=50 runtime=300 time_based=1 bs=1048576 rw=randrw verify=crc32c-intel verify_dump=1 verify_fatal=1 verify_pattern=8 verify_backlog=1 iodepth=16 max_latency=120s size=21474836480 ramp_time=6s end_fsync=1 write_bw_log=/tmp/fio_AVIiUF/r11i1_job0_zZRgcjyC.IO_TEST/r11i1_job0_zZRgcjyC.IO_TEST write_lat_log=/tmp/fio_AVIiUF/r11i1_job0_zZRgcjyC.IO_TEST/r11i1_job0_zZRgcjyC.IO_TEST write_iops_log=/tmp/fio_AVIiUF/r11i1_job0_zZRgcjyC.IO_TEST/r11i1_job0_zZRgcjyC.IO_TEST log_avg_msec=10000 ioengine=libaio

# cat /etc/exports # See the exports(5) manpage for a description of the syntax of this file. # This file contains a list of all directories that are to be exported to # other computers via NFS (Network File System). # This file used by rpc.nfsd and rpc.mountd. See their manpages for details # on how make changes in this file effective.

/shares/universal_share *(rw,no_subtree_check,async,wdelay,root_squash)

NFS version 4

[sitsofe]

@rustyscottweber You're running fio as root but you say your share NFS share has root_squash which remaps root to a user that typically doesn't have permissions? Does using

touch /tmp/mount_192_168_201_1_universal_share/r11i1_job0_KOyqHgFS.IO_TEST

as root have the same problem? If so this is not an fio issue...

[rustyscottweber]

The test file can be touched and fio has no problem laying down the test files. I can even lay down the test files using fio, remount with the no_root_sqaush flag and then run the traffic test which will then succeed with flying colors. Per the documentation: root_squash prevents remote root users from having superuser (root) privileges on remote NFS-mounted volumes by remapping the root user to a different UID on the NFS server. However, I should still have enough access to the file on the client which has the nfs share mounted to read and write the file in an asynchronous fashion (I'm not sure what other permissions would be needed). It's possible that this is a bug in the NFS server, but before I kick it over to the NFS devs, I'd like to know what method, exactly, is failing and whether it's the NFS client or the NFS server at fault.

[axboe]

Can you try and lay the files down, and then do a fio run? That should fail immediately. For that run, maybe do:

strace -o xx -f ./fio bla.ini

and attach xx? It looks like it's throwing EPERM on open, I wonder if it's one of the flags being passed in. That would normally yield EINVAL, but it seems weird. Maybe the full trace will shine some light on what is going on here.

[rustyscottweber]

strace_output.zip uploaded

[axboe]

Thanks for sending the strace! It looks like we're getting EACCES on open of those files. We open it fine from the main thread, and get EACCES from the client:

Main thread:

26167 openat(AT_FDCWD, "/tmp/mount_192_168_200_1_universal_share/r11i1_job1_ReTJPLEh.IO_TEST", O_WRONLY|O_CREAT|O_TRUNC, 0644) = 6

Client IO thread:

26371 openat(AT_FDCWD, "/tmp/mount_192_168_200_1_universal_share/r11i1_job1_ReTJPLEh.IO_TEST", O_RDWR|O_CREAT|O_DIRECT, 0600 <unfinished ...>
26371 <... openat resumed>)             = -1 EACCES (Permission denied)

Not sure why that is, doesn't really seem like a fio issue to be honest.

[rustyscottweber]

Filed a bug on the NFS side of things here: https://bugzilla.linux-nfs.org/show_bug.cgi?id=363 @axboe , can we keep this ticket open until I get a resolution from the NFS devs?

[axboe]

Sure, no problem.