[Only affects wings 1.11.9 and up] Unable to mount an NFS "folder" inside the pterodactyl server volume.

[llek1000]

Current Behavior

After mounting an NFS share into a server's volume (for example /var/lib/pterodactyl/volumes/volume_id/data) and after starting the server, the server throws this error: Error Event [68d61a4e-0836-4694-99c0-6094e2819970]: failed to chown root server directory during pre-boot process: server/filesystem: chown: failed to chown during walk function: stat 0c89372c-9a08-4cee-b516-7dd0d4db3bca/part.602: file does not exist

I tried on two separate nodes and got the same result.

Expected Behavior

Mount an NFS share into a server's volume (for example /var/lib/pterodactyl/volumes/volume_id/data) start the server, the server will change its permissions and everything works fine - this was the behaviour before wings version 1.11.9

Steps to Reproduce

1. Create a server - it does not matter which egg or docker image.
2. Mount an NFS share into this servers volume
3. Start the server

Panel Version

1.11.5

Wings Version

1.11.11

Games and/or Eggs Affected

No response

Docker Image

No response

Error Logs

https://ptero.co/igoselytov

Is there an existing issue for this?

• [X] I have searched the existing issues before opening this issue.
• [X] I have provided all relevant details, including the specific game and Docker images I am using if this issue is related to running a server.
• [X] I have checked in the Discord server and believe this is a bug with the software, and not a configuration issue with my specific system.

[ptoal]

I can confirm this. I have an NFS4 mounted disk at /var/srv/gstorage1/wings1 that is owned by my pterodactyl:pterodactyl user. When attempting to start any server with 1.11.11, I get the same error. Reverting back to 1.11.8 works fine.

I haven't looked at the code yet, but my first guess is that wings is doing something with privileges that is changing its effective user id, and the NFS server is denying access.

[llek1000]

Yes, I've also downgraded the node that runs the server with the mount. The node is private anyways so it does not really matter for now.

I do not think it's something to do with the user id, although I am not an expert by any stretch of imagination, so please excuse me if I'm wrong - since I do not really know how the internal processes of Wings work. However, I've spun up a new NFS4 share that allowed any user from the IP of the node to read/write and I got the same result.

From what I've seen in the log: WARN: [Mar 22 17:36:51.772] failed to determine root fs directory size error=server/filesystem: directorysize: failed to walk directory: walkdirat err: stat 0c89372c-9a08-4cee-b516-7dd0d4db3bca/part.602: file does not exist root=/var/lib/pterodactyl/volumes/a8c306f0-a28d-47c2-aeec-147b7c90df30 it cannot find the file 0c89372c-9a08-4cee-b516-7dd0d4db3bca/part.602 which is interesting, because the NFS4 share is mounted in 0c89372c-9a08-4cee-b516-7dd0d4db3bca/data/ and not in the root of the volume.

Also, Wings apparently has a problem with calculating the volume size:

WARN: [Mar 22 17:36:51.772] failed to determine root fs directory size error=server/filesystem: directorysize: failed to walk directory: walkdirat err: stat 0c89372c-9a08-4cee-b516-7dd0d4db3bca/part.602: file does not exist root=/var/lib/pterodactyl/volumes/a8c306f0-a28d-47c2-aeec-147b7c90df30

Stacktrace:
stat 0c89372c-9a08-4cee-b516-7dd0d4db3bca/part.602: file does not exist
walkdirat err
github.com/pterodactyl/wings/server/filesystem.(*Filesystem).DirectorySize.func1
	github.com/pterodactyl/wings/server/filesystem/disk_space.go:170
github.com/pterodactyl/wings/internal/ufs.(*UnixFS).walkDir
	github.com/pterodactyl/wings/internal/ufs/walk_unix.go:59
github.com/pterodactyl/wings/internal/ufs.(*UnixFS).walkDir
	github.com/pterodactyl/wings/internal/ufs/walk_unix.go:76
github.com/pterodactyl/wings/internal/ufs.(*UnixFS).walkDir
	github.com/pterodactyl/wings/internal/ufs/walk_unix.go:76
github.com/pterodactyl/wings/internal/ufs.(*UnixFS).walkDir
	github.com/pterodactyl/wings/internal/ufs/walk_unix.go:76
github.com/pterodactyl/wings/internal/ufs.(*UnixFS).walkDir
	github.com/pterodactyl/wings/internal/ufs/walk_unix.go:76
github.com/pterodactyl/wings/internal/ufs.(*UnixFS).walkDir
	github.com/pterodactyl/wings/internal/ufs/walk_unix.go:76
github.com/pterodactyl/wings/internal/ufs.(*UnixFS).walkDir
	github.com/pterodactyl/wings/internal/ufs/walk_unix.go:76
github.com/pterodactyl/wings/internal/ufs.(*UnixFS).WalkDirat
	github.com/pterodactyl/wings/internal/ufs/walk_unix.go:33
github.com/pterodactyl/wings/server/filesystem.(*Filesystem).DirectorySize
	github.com/pterodactyl/wings/server/filesystem/disk_space.go:168
github.com/pterodactyl/wings/server/filesystem.(*Filesystem).updateCachedDiskUsage
	github.com/pterodactyl/wings/server/filesystem/disk_space.go:147
github.com/pterodactyl/wings/server/filesystem.(*Filesystem).DiskUsage
	github.com/pterodactyl/wings/server/filesystem/disk_space.go:113
github.com/pterodactyl/wings/server/filesystem.(*Filesystem).HasSpaceAvailable
	github.com/pterodactyl/wings/server/filesystem/disk_space.go:68
github.com/pterodactyl/wings/server/filesystem.(*Filesystem).HasSpaceErr
	github.com/pterodactyl/wings/server/filesystem/disk_space.go:53
github.com/pterodactyl/wings/server.(*Server).onBeforeStart
	github.com/pterodactyl/wings/server/power.go:194
github.com/pterodactyl/wings/server.(*Server).HandlePowerAction
	github.com/pterodactyl/wings/server/power.go:131
github.com/pterodactyl/wings/router/websocket.(*Handler).HandleInbound
	github.com/pterodactyl/wings/router/websocket/websocket.go:363
github.com/pterodactyl/wings/router.getServerWebsocket.func3
	github.com/pterodactyl/wings/router/router_server_ws.go:85
runtime.goexit
	runtime/asm_amd64.s:1650

server/filesystem: directorysize: failed to walk directory

[parkervcp]

This probably has to do with the fact that we are using the unix file stuff now as a change from the old filepath.

I am going to assume they are treated differently because it's externally mounted and the io package doesn't know how to handle the mount properly.

[matthewpi]

Where exactly on the filesystem are you mounting to?

I think I might know what the issue is.

[llek1000]

Where exactly on the filesystem are you mounting to?

I think I might know what the issue is.

I am mounting exactly to /var/lib/pterodactyl/volumes/volume_id/data, of course, where "volume_id" is replaced by the actual volume_id.

Edit: Just in case it helps, I am using this mount under the minio egg, where I am mounting the data folder inside the pterodactyl volume to my NAS with NFSv4. Oh, and I do not have Squash enabled on my NFS server.

[Hmmmz]

I have a similar problem with this error that "file not exist" when checking file permission while starting on a NFS share with the new Wings version but I mount to "/var/lib/pterodactyl" instead, any ideas?

[llek1000]

Hello, just checking in, any updates?

[simonmicro]

Hey, same problem here - just killed all my servers as I rolled out the update via Ansible. This also seemed to cause a lot of corrupt backups, idk how this came. Stupid mistake from my side.

[DerMistkaefer]

We have the same problem with servers/volumes that are created dynamically. We archive the volume dirs and then transfer them to the NFS share. The issue occurs sometime, so It's hard to analyze.

When I then restart wings everything works fine. So it seems there is a problem with new generated volume dirs. Like a cache or something.

So I think the problem is that when the server and the wings ufs filesystem instance is created that there is no volume dir exist. So then when we unzip the correct volume or for other the nfs mount get created there is another memory reference and the inital created ufs filesystem isn't exist anymore. So the link is broken. https://github.com/pterodactyl/wings/blob/develop/server/filesystem/filesystem.go#L41

I think the memory reference is the https://github.com/pterodactyl/wings/blob/develop/internal/ufs/fs_unix.go#L46 and need to be recreated if the basePath get changed. Or is not exist anymore.

I hope this helps, but I don't know the unix package and if is relevant if we have Openat2

[DerMistkaefer]

What worked for me, I don't delete the volume folder anymore. So I only delete the content of the volume folder, then unzip the files from the nfs share.

So the folder memory link don't get destroy.

[azra026]

https://pastebin.com/CkcDH9hj

Still an issue with the latest release, only 1.11.8 works.

[NobodyCentral]

I'm also having this issue. I had wings version v1.11.11. I have an NFS share mounted to /mnt/ptero and changed the configs for wings to use that as its storage and when I try to start some but not all of my game servers it gets this error, but the moment I downgraded wings to version 1.11.8 it magically started working as normal.

[WouterGritter]

I'm running into the same issue, however my NFS share is not inside a server's volume directory. For me, the pterodactyl volumes folder is located on an NFS share.

I downgraded wings to v1.11.8 and it works for now. Really a shame that the current version is 1.11.13 and I have to downgrade so far. Is it really that uncommon to run wings volumes on an external NFS share?

[matthewpi]

https://github.com/pterodactyl/wings/commit/8744e64f1d888921b6f7cac1ebacc31d86e48cd4