Issues icon indicating copy to clipboard operation
Issues copied to clipboard
verified publisher icon OctopusDeploy

Octopus Server Linux Docker image permissions change on upgrade to 2024.1

Open borland opened this issue 1 year ago • 1 comments

Severity

Blocking one known customer

Version

2024.1.12413

Latest Version

I could reproduce the problem in the latest build

What happened?

Customer is attempting to upgrade their docker image to the 2024.1.12413 official Octopus one (https://hub.docker.com/layers/octopusdeploy/octopusdeploy/2024.1.12413/images/sha256-4508e14315ea70d850c50ee77295cecbe75b64fa6d878150de1947cb338884d6?context=explore). They seemed to have run into permissions issues, and it looks like the user ID used in the 2024 ones is now 1999. In pre-2024, this was 999.

The UID value affects permissions on filesystem volumes that are mapped into the container. Files and directories created with an earlier version will be owned by UID 999, and may be unreadable or unwritable after upgrade when the server is now running as 1999

Reproduction

  • Run the octopusdeploy linux container, version 2023.4, with mapped filesystem volumes per our docs
  • Perform some activity that writes to the filesystem, such as uploading a package
  • Shut down the 2023.4 container
  • Run the octopusdeploy linux container, version 2024.1, with the same mapped filesystem volumes and DB connection as earlier.

Permissions related errors will be observed, relating to the mapped filesystem volumes

Error and Stacktrace

Exception appending to log with correlationId ScheduledTasks_ProcessAutoDeployments
System.UnauthorizedAccessException: Access to the path '/taskLogs/scheduledtasks_processautodeployments.txt' is denied.
---> System.IO.IOException: Permission denied
   --- End of inner exception stack trace ---
   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirError)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, UnixFileMode openPermissions, Int64& fileLength, UnixFileMode& filePermissions, Boolean failForSymlink, Boolean& wasSymlink, Func`4 createOpenException)
   at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
   at Octopus.Shared.Util.OctopusPhysicalFileSystem.OpenFile(String path, FileMode mode, FileAccess access, FileShare share) in ./source/Octopus.Shared/Util/OctopusPhysicalFileSystem.cs:line 243
   at Octopus.Core.Orchestration.Logging.Processing.AppendToLogFile.InitializeInternal() in ./source/Octopus.Core/Orchestration/Logging/Processing/AppendToLogFile.cs:line 44
   at Octopus.Server.Orchestration.Logging.FileBasedWriterBuilder.BuildWriter(RootCorrelationId rootCorrelationId) in ./source/Octopus.Server/Orchestration/Logging/FileBasedWriterBuilder.cs:line 54
   at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
   at Octopus.Server.Orchestration.Logging.ServerLogStorage.Append(CorrelationId correlationId, ActivityLogEntry entry) in ./source/Octopus.Server/Orchestration/Logging/ServerLogStorage.cs:line 85
"HTTP" "GET" to "<redacted>""/api/Spaces-1/projects/Projects-1/deploymentprocesses" "completed" with 200 in 00:00:00.0182972 (18ms) by "admin"
Exception appending to log with correlationId ScheduledTasks_ProcessAutoDeployments
System.UnauthorizedAccessException: Access to the path '/taskLogs/scheduledtasks_processautodeployments.txt' is denied.
---> System.IO.IOException: Permission denied
   --- End of inner exception stack trace ---
   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirError)
"HTTP" "GET" to "<redacted>""/api/Spaces-1/projects/Projects-1/channels" "completed" with 304 in 00:00:00.0111825 (11ms) by "admin"
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, UnixFileMode openPermissions, Int64& fileLength, UnixFileMode& filePermissions, Boolean failForSymlink, Boolean& wasSymlink, Func`4 createOpenException)
   at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
   at Octopus.Shared.Util.OctopusPhysicalFileSystem.OpenFile(String path, FileMode mode, FileAccess access, FileShare share) in ./source/Octopus.Shared/Util/OctopusPhysicalFileSystem.cs:line 243
   at Octopus.Core.Orchestration.Logging.Processing.AppendToLogFile.InitializeInternal() in ./source/Octopus.Core/Orchestration/Logging/Processing/AppendToLogFile.cs:line 44
   at Octopus.Server.Orchestration.Logging.FileBasedWriterBuilder.BuildWriter(RootCorrelationId rootCorrelationId) in ./source/Octopus.Server/Orchestration/Logging/FileBasedWriterBuilder.cs:line 54
   at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
   at Octopus.Server.Orchestration.Logging.ServerLogStorage.Append(CorrelationId correlationId, ActivityLogEntry entry) in ./source/Octopus.Server/Orchestration/Logging/ServerLogStorage.cs:line 85

More Information

No response

Workaround

No response

borland avatar Apr 18 '24 09:04 borland

Note on behalf of Octopus R&D.

The UID/GID change occurred because while we were working on upgrading Octopus Server to use .NET 8 for the 2024.1 release, we internally also changed the base docker image from Debian 11 to Debian 12.

The Octopus Server container image before 2024.1 uses UID/GID of 999. This is a bug in our container configuration; UID/GID values below 1000 are reserved for system use, however in Debian 11 and below, the 999 value is not used by any system processes and no problems are observed.

Debian 12 runs the systemd-journal process with GID 999 which conflicts with Octopus Server's configured value. We needed to change the Octopus Server configuration to something other than 999 to proceed with our internal Debian 12 testing. We picked 1999 as a replacement.

We later changed back to Debian 11, and our official Octopus Server container images for 2024.1 ship on 11 rather than 12, but we decided to keep the UID 1999 change to pre-empt the future Debian change. With the benefit of hindsight, this was not the correct decision and we have decided to revert the UID/GID to 999 so as not to unduly affect customers.

Note: At a future point we will need to upgrade the base image to Debian 12 or newer, and the UID/GID will change away from 999, This will again be a breaking change for customers, however we plan to provide better communication and advance warning of it next time.

borland avatar Apr 18 '24 10:04 borland

Release Note: OctopusDeploy linux docker container UID and GID restored to 999 value used by 2023.4 and earlier versions

octoreleasebot avatar Apr 30 '24 04:04 octoreleasebot

:tada: The fix for this issue has been released in:

Release stream Release
2024.1 2024.1.12528
2024.2 2024.2.6009

Octobob avatar Apr 30 '24 04:04 Octobob

:tada: The fix for this issue has been released in:

Release stream Release
2024.1 2024.1.12528
2024.2 2024.2.6009
2024.3+ all releases

Octobob avatar May 16 '24 10:05 Octobob