Using CIFS mounts for task log storage can cause them to be unable to be written

[liam-mackie]

trafficstars

Severity

Can cause deployment failures

Version

Any version

Latest Version

Not applicable

What happened?

In certain circumstances, Octopus Server is unable to write log files to volumes mounted using CIFS.

This only occurs when using the following CIFS module versions (see the translation to kernel versions here)

• 2.24 - 2.36
• 2.39+

The issue stems from this file locking issue-,CIFS%20details,-Up%20to%20Linux), specifically:

       Since Linux 5.5, flock() locks are emulated with SMB byte-range
       locks on the entire file.  Similarly to NFS, this means that
       fcntl(2) and flock() locks interact with one another.  Another
       important side-effect is that the locks are not advisory anymore:
       any IO on a locked file will always fail with EACCES when done
       from a separate file descriptor.  This difference originates from
       the design of locks in the SMB protocol, which provides mandatory
       locking semantics.

Due to the interaction of how .NET + Linux + SMB do file locking, any file opened on a Linux kernel >= 5.5 on SMB with FileShare.ReadWrite will block all writes from other FDs. This is not a bug, rather a poorly discoverable limitation.

This means that when Server opens a read stream to serve a user viewing the file, it locks the file exclusively, and can't write at the same time.

Reproduction

1. Run server on Linux (with an affected CIFS version) with the task logs being stored on a CIFS mount (for example, Azure Files)
2. Run a long-running task that logs frequently
3. Open the task log view in a web browser
4. Observe server logs to see the UnauthorizedAccessException when writing

Error and Stacktrace

System.UnauthorizedAccessException: Access to the path '<log-location>' is denied.
 ---> System.IO.IOException: Permission denied
   --- End of inner exception stack trace ---
   at System.IO.FileStream.WriteNative(ReadOnlySpan`1 source)
   at System.IO.FileStream.FlushWriteBuffer()
   at System.IO.FileStream.Dispose(Boolean disposing)
   at System.IO.Stream.Close()
   at System.IO.StreamWriter.CloseStreamFromDispose(Boolean disposing)
   at System.IO.StreamWriter.Dispose(Boolean disposing)
   at Octopus.Server.Orchestration.Logging.Processors.AppendToLogFile.Dispose() in ./source/Octopus.Server/Orchestration/Logging/Processors/AppendToLogFile.cs:line 98
   at Octopus.Server.Orchestration.Logging.ServerLogWriter.Dispose() in ./source/Octopus.Server/Orchestration/Logging/ServerLogWriter.cs:line 102
   at Octopus.Server.Orchestration.Logging.ServerLogStorage.SafeDisposeWriter(ServerLogWriter writer) in ./source/Octopus.Server/Orchestration/Logging/ServerLogStorage.cs:line 226",

More Information

This was the subject of an internal incident - #incident-20231206-blithe-tree

Workaround

You can workaround this issue by setting the nobrl mount option on the CIFS mount, but this may lead to unintended consequences if running in a HA setup.

[LukeButters]

Potentially related .net code