Upgrading tentacles in a specific environment blocks deployments / runbook runs for all environments

[BobJWalker]

Team

• [X] I've assigned a team label to this issue

Severity

Any customer who upgrades will have all their deployments blocked during any tentacle upgrades. The bigger the customer the worse it is.

Version

2021.2 but based on git history this might've been going on for months and months.

Latest Version

I could reproduce the problem in the latest build

What happened?

You can choose to upgrade all the tentacles in a specific environment rather than all the tentacles.

That triggers a POST to the /tasks endpoint with the following payload:

{
  "Arguments": {
    "SpaceId": "Spaces-1",
    "RestrictedTo": "DeploymentTargets",
    "EnvironmentId": "Environments-1"
  },
  "Description": "Upgrade Tentacles in Development",
  "Name": "Upgrade"
}

We can see in the log this message:

That message indicates the code got to this line: https://github.com/OctopusDeploy/OctopusDeploy/blob/master/source/Octopus.Server/Orchestration/ServerTasks/Upgrade/UpgradeTaskController.cs#L104

The upgrade is for all tentacles in a specific environment. However, when I run a runbook in my Test environment, it is blocked by that upgrade task:

In looking at the database we can see why this is happening. The EnvironmentID column is populated for the runbook run, but not the upgrade task (even though it was limited by environments). The upgrade environment ID is stored in the arguments JSON.

In diving into the code, the function IsTaskBlockedByCurrentlyExecutingTasks on the TaskQueue object is comparing Task.EnvironmentId with UpgradeTask.EnvironmentId, not UpgradeTask.Arguments.EnvironmentId.

When I attempt to change the payload to send in the EnvironmentId in the root object, that additional environment id is ignored.

{
  "Arguments": {
    "SpaceId": "Spaces-1",
    "RestrictedTo": "DeploymentTargets",
    "EnvironmentId": "Environments-1"
  },
  "Description": "Upgrade Tentacles in Development",
  "EnvironmentId": "Environments-1"
  "Name": "Upgrade"
}

Reproduction

Steps to reproduce:

Prep-work: Create a runbook to run a "Hello-World" script. Configure it to run in any environment.

1. Install an older version of the tentacle (I was choosing 6.0.0)
2. Do a health check to ensure that tentacle is showing up as requiring an upgrade
3. Go to the Infrastructure -> Environments and click the overflow menu (...) on the environment with the tentacle to upgrade.
4. Choose the option Upgrade 1 Tentacle in this Environment to start the upgrade.
5. While the upgrade is running run the runbook you created in earlier in any other environment. You'll see it block.

Error and Stacktrace

No response

More Information

Octopus Code I looked at:

• Octopus.Server.Orchestration.TaskQueue.TaskQueue
• Octopus.Server.Orchestration.ServerTasks.Upgrade.UpgradeTaskController
• Octopus.Server.Orchestration.ServerTasks.HealthCheck.MachineTaskController

Workaround

Wait to schedule tentacle upgrades during off-hours. For large companies with 100s of tentacles running 24/7 this is incredibly difficult and they have to bite the bullet and block everyone.

[tothegills]

Similar issue: https://github.com/OctopusDeploy/Issues/issues/3535

[BobJWalker]

@tothegills it is similar, but in this case, there already is code in place with the intention of allowing tentacle upgrades and deployments to occur at the same time for different environments. It just isn't working as expected.

[patrick-smergut-octopus]

Another report here [internal]: https://octopus.zendesk.com/agent/tickets/80917

[paraicoceallaigh]

Another report here [private link]: https://octopus.zendesk.com/agent/tickets/84752

[Clare-Octopus]

Another report (Internal ticket) - https://octopus.zendesk.com/agent/tickets/113079