Kubernetes API Deployment Targets step section not being ignored when step is run on Kubernetes Agent Targets and fails deployments with 'docker: command not found'

[patrick-smergut-octopus]

Severity

Blocks deployments for anyone who has configured this section and is running against a K8s Agent target

Version

2024.4.6608

Latest Version

None

What happened?

If the Kubernetes API Deployment Targets section of a step is configured to pull a docker image from a feed, this will not be ignored when run on behalf of a Kubernetes Agent target. The script pod will attempt to pull the specified docker image, but it will fail since the docker command won't be found.

Because this deployment is being made to a Kubernetes Agent target, the Kubernetes API Deployment Targets section should be ignored.

Reproduction

1. Configure a Kubernetes Agent target.
2. In a project deployment process, use the Configure and apply Kubernetes resources step with some resources (e.g. nginx) and set the Kubernetes API Deployment Targets section to use a docker feed with a referenced container
3. Attempt to deploy the resources to the Kubernetes Agent target
4. See docker: command not found error during package acquisition on the Kubernetes Agent for the step, as well as docker feed authorization failure

Error and Stacktrace

20:31:18   Verbose  |       Unable to extract labels from Image manifest so they will not be included in the deployment variables
                    |       Authorization to `https://registry-1.docker.io/v2/octopuslabs/k8s-wokertools/manifests/latest` failed.
                    |       Octopus.Core.Packages.Docker.DockerOAuthException: Authorization to `https://registry-1.docker.io/v2/octopuslabs/k8s-wokertools/manifests/latest` failed.
                    |       at Octopus.Core.Packages.Docker.DockerRegistryOAuthClient.GetInternal(Uri url, ICredentials credentials, CancellationToken cancellationToken) in ./source/Octopus.Core/Packages/Docker/DockerRegistryOAuthClient.cs:line 143
                    |       at Octopus.Core.Packages.Docker.DockerRegistryOAuthClient.Get(Uri url, ICredentials credentials, CancellationToken cancellationToken) in ./source/Octopus.Core/Packages/Docker/DockerRegistryOAuthClient.cs:line 114
                    |       at Octopus.Core.Packages.Docker.DockerAdditionalVariablesContributor.ExtractManifestLabels(DockerFeed feed, String packageId, IVersion version) in ./source/Octopus.Core/Packages/Docker/DockerAdditionalVariablesContributor.cs:line 70
                    |       Octopus.Server version 2024.4.6608 (2024.4.6608)
20:31:25   Fatal    |       The remote script failed with exit code 1
20:31:25   Verbose  |       at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.SuccessArbitrator.ThrowIfNotSuccessful(IActionHandlerResult result) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/ActionDispatch/SuccessArbitrator.cs:line 25
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.AdHocActionDispatcher.Dispatch(ExecutionsMachine machine, ActionHandlerInvocation actionHandler, ITaskLog taskLog, CancellationToken cancellationToken, VariableCollection variables) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/ActionDispatch/AdHocActionDispatcher.cs:line 80
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.Acquire.PackageAcquisitionExecutionHandlers.Invoke(ActionHandlerInvocation invocation, ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/Steps/Acquire/PackageAcquisitionExecutionHandlers.cs:line 203
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.Acquire.PackageAcquisitionExecutionHandlers.DownloadPackage(String packageId, IVersion version, IFeed feed, ITaskLog taskLog, CancellationToken cancellationToken, PackageCachePolicy packageCachePolicy) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/Steps/Acquire/PackageAcquisitionExecutionHandlers.cs:line 144
                    |       at Octopus.Server.Orchestration.Targets.Common.V2.StagingPackageDownloader.DownloadPackage(PackageIdentity packageIdentity, FROZEN_IDeploymentPackageDownloader deploymentPackageDownloader, IPackageAcquisitionService execs, IPackageFeed packageFeed, ITaskLog logBlock, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/Targets/Common/V2/StagingPackageDownloader.cs:line 19
                    |       at Octopus.Server.Orchestration.Targets.Common.V2.DownloadPackageOnAgentSteps.GetDownloadedPackageActionResult(PackageIdentity packageIdentity, FROZEN_IDeploymentPackageDownloader deploymentPackageDownloader, IPackageAcquisitionService execs, ITaskLog taskLog, IPackageFeed packageFeed, IEnumerable`1 additionalVariables, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/Targets/Common/V2/DownloadPackageOnAgentSteps.cs:line 74
                    |       at Octopus.Server.Orchestration.Targets.Common.V2.DownloadPackageOnAgent.Execute(PackageIdentity packageIdentity, FROZEN_IDeploymentPackageDownloader deploymentPackageDownloader, IPackageAcquisitionService execs, ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/Targets/Common/V2/DownloadPackageOnAgent.cs:line 35
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.Acquire.AcquireMachinePackageTask.StagePackage(ITaskLog stepTaskLog, ITaskLog deploymentTargetTaskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/Steps/Acquire/AcquireMachinePackageTask.cs:line 219
                    |       at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.Acquire.AcquireMachinePackageTask.<>c__DisplayClass26_0.<<Acquire>b__0>d.MoveNext() in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/Steps/Acquire/AcquireMachinePackageTask.cs:line 136
                    |       --- End of stack trace from previous location ---


================Further down in the log==================

20:31:24   Verbose  |         Kubernetes Script Pod started
20:31:24   Verbose  |         Started Calamari PID 13
20:31:24   Verbose  |         Calamari Version: 28.2.0
20:31:24   Verbose  |         Environment Information:
20:31:24   Verbose  |         OperatingSystem: Unix 5.15.0.1073
20:31:24   Verbose  |         OsBitVersion: x64
20:31:24   Verbose  |         Is64BitProcess: True
20:31:24   Verbose  |         CurrentUser: root
20:31:24   Verbose  |         MachineName: octopus-script-wixiafyr8ushtyxrn2oza
20:31:24   Verbose  |         ProcessorCount: 2
20:31:24   Verbose  |         CurrentDirectory: /octopus/Work/wIXiAfyR8USHtYxRn2ozA
20:31:24   Verbose  |         TempDirectory: /tmp/
20:31:24   Verbose  |         HostProcess: Calamari (13)
20:31:24   Verbose  |         Feed type provided `Docker` using DockerImagePackageDownloader
20:31:24   Verbose  |         Setting Proxy Environment Variables
20:31:24   Verbose  |         The Kubernetes Pod 'octopus-script-wixiafyr8ushtyxrn2oza' is in the 'Running' phase
20:31:24   Verbose  |         Bash Environment Information:
20:31:24   Verbose  |         OperatingSystem: Linux octopus-script-wixiafyr8ushtyxrn2oza 5.15.0-1073-azure #82-Ubuntu SMP Mon Sep 2 11:36:34 UTC 2024 x86_64 GNU/Linux
20:31:24   Verbose  |         CurrentUser: root
20:31:24   Verbose  |         HostName: octopus-script-wixiafyr8ushtyxrn2oza
20:31:24   Verbose  |         ProcessorCount: 2
20:31:24   Verbose  |         CurrentDirectory: /octopus/Work/wIXiAfyR8USHtYxRn2ozA
20:31:24   Verbose  |         TempDirectory: /tmp
20:31:24   Verbose  |         HostProcessID: 24
20:31:24   Info     |         /octopus/Work/wIXiAfyR8USHtYxRn2ozA/Octopus.DockerLogin.sh: line 19: docker: command not found
20:31:24   Info     |         Login Failed
20:31:24   Error    |         /octopus/Work/wIXiAfyR8USHtYxRn2ozA/Octopus.DockerLogin.sh: line 8: docker: command not found
20:31:24   Error    |         /octopus/Work/wIXiAfyR8USHtYxRn2ozA/Octopus.DockerLogin.sh: line 13: docker: command not found
20:31:24   Verbose  |         Waiting 0s before attempting the download from the external feed again.
20:31:24   Verbose  |         Setting Proxy Environment Variables
20:31:24   Verbose  |         Bash Environment Information:
20:31:24   Verbose  |         OperatingSystem: Linux octopus-script-wixiafyr8ushtyxrn2oza 5.15.0-1073-azure #82-Ubuntu SMP Mon Sep 2 11:36:34 UTC 2024 x86_64 GNU/Linux
20:31:24   Verbose  |         CurrentUser: root
20:31:24   Verbose  |         HostName: octopus-script-wixiafyr8ushtyxrn2oza
20:31:24   Verbose  |         ProcessorCount: 2
20:31:24   Verbose  |         CurrentDirectory: /octopus/Work/wIXiAfyR8USHtYxRn2ozA
20:31:24   Verbose  |         TempDirectory: /tmp
20:31:24   Verbose  |         HostProcessID: 70
20:31:24   Error    |         /octopus/Work/wIXiAfyR8USHtYxRn2ozA/Octopus.DockerLogin.sh: line 8: docker: command not found
20:31:24   Error    |         /octopus/Work/wIXiAfyR8USHtYxRn2ozA/Octopus.DockerLogin.sh: line 13: docker: command not found
20:31:24   Info     |         /octopus/Work/wIXiAfyR8USHtYxRn2ozA/Octopus.DockerLogin.sh: line 19: docker: command not found
20:31:24   Info     |         Login Failed
20:31:24   Error    |         Failed to download package octopuslabs/k8s-wokertools vlatest from feed: 'https://index.docker.io'
20:31:24   Error    |         Unable to log in Docker registry
20:31:24   Verbose  |         Kubernetes Script Pod completed
20:31:24   Verbose  |         EOS-075CD4F0-8C76-491D-BA76-0879D35E9CFE<<>>1
20:31:24   Verbose  |         Marking 'wIXiAfyR8USHtYxRn2ozA' as completed with exit code: '1'
20:31:25   Verbose  |         The Kubernetes Pod 'octopus-script-wixiafyr8ushtyxrn2oza' is in the 'Failed' phase
20:31:25   Verbose  |         Exit code: 1

More Information

This behavior was initially observed in a Kustomize step: Internal link.

Workaround

Change the Kubernetes API Deployment Targets section of the step to run directly on the worker: