System.IO.IOException: Permission denied on SimpleFileDb with 1.25.3

[AntiD2ta]

Description I got the following error while running Nethermind with Prysm in a bare metal server on mainnet:

Failed to store data in /data/execution/nethermind_db/mainnet/discoveryNodes/SimpleFileDb.db System.UnauthorizedAccessException: Access to the path '/data/execution/nethermind_db/mainnet/discoveryNodes/SimpleFileDb.db' is denied.

Prysm was broken and throwing cryptic (not informative) panics. This after a week of the node started to sync.

I suspect the pod (this was on K8S) ran out of space and that's why Prysm panicked, but I can't confirm it and Nethermind was not throwing its usual No space on device error.

Nethermind keeps running but sending those exceptions from time to time.

Expected behavior Nethermind docker image user has root access and shouldn't throw permission issues.

Desktop: Please provide the following information regarding your setup:

• Operating System: Ubuntu
• Version: 1.25.3
• Installation Method: Docker Image
• Consensus Client: Prysm v4.2.0

Additional Context Snap sync was also stuck in the same percent.

Logs Nethermind's logs:

30 Jan 12:13:09 | Failed to store data in /data/execution/nethermind_db/mainnet/discoveryNodes/SimpleFileDb.db System.UnauthorizedAccessException: Access to the path '/data/execution/nethermind_db/mainnet/discoveryNodes/SimpleFileDb.db' is denied.
 ---> System.IO.IOException: Permission denied
   --- End of inner exception stack trace ---
   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirError)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, UnixFileMode openPermissions, Int64& fileLength, UnixFileMode& filePermissions, Boolean failForSymlink, Boolean& wasSymlink, Func`4 createOpenException)
   at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
   at Nethermind.Db.SimpleFilePublicKeyDb.CommitBatch() in /_/src/Nethermind/Nethermind.Db/SimpleFilePublicKeyDb.cs:line 113