Uploads fails to immutable fileset

[janfrode]

Environment info

• NooBaa Version: 5.15.4-20240704.el8.x86_64
• Platform: Storage Scale 5.2.1.0 / RHEL8 / x86-64

Actual behavior

1. When trying to upload objects to an immutable fileset, it fails with error account AccessDenied.

Expected behavior

1. Uploads should succeed.

Steps to reproduce

# mms3 account create s3account --uid 1001 --gid 1001 --newBucketsPath "/ibm/fs1/s3account/"

Account s3account created successfully
 Access Key             Secret Key
 ---------              ----------
w0wDKyXu4jDvXWnwNIL1    dZ+lHgLJ8F0mNgnGvFpQtQTkAO9X5BviEUrAXvuy

# mmcrfileset fs1 immutable --inode-space new
# mmchfileset fs1 immutable --iam-mode noncompliant
# mmlinkfileset fs1 immutable -J /ibm/fs1/immutable
# chown 1001:1001 /ibm/fs1/immutable
# mms3 bucket create immutable --accountName s3account --filesystemPath /ibm/fs1/immutable

# aws --no-verify-ssl  --endpoint-url='https://10.33.3.111:6443' s3 cp /etc/passwd s3://immutable
upload failed: ../etc/passwd to s3://immutable/passwd An error occurred (AccessDenied) when calling the PutObject operation: Access Denied

More information - Screenshots / Logs / Other output

This is most likely caused how the file is created or renamed after upload. If we point GPFSDLPATH to a non-existent path (to disable the libgpfs.so integration), then the uploads work. I suspect the problem is either that O_TMPFILE doesn't work with immutable fileset, or that the uploaded file is created with too strict permissions initially -- which would trigger it to become immutable, and that it then fails to relink to the final destination.

GPFS immutability is automatically enabled for a file if we remove write access:

# touch testfile
# ls -l testfile
-rw-r--r--. 1 root          root             0 Aug 30 16:31 testfile
# mmlsattr -L testfile |grep -i immu
immutable:            no

# chmod u-w testfile
# mmlsattr -L testfile |grep -i immu
immutable:            yes

I suspect it's this functionality that's kicking in during the linking of the O_TMPFILE somehow..

Additionally, I tested keeping the top-level .noobaa-nsfs directory on a non-immutable fileset, and just having the directory we uploaded to be an immutable directory one level further down, but this also failed the same way. So it's not the .noobaa-nsfs tmp-files that's causing the access denied.