Deployment timing out when retention is running

[veochen-octopus]

Team

• [X] I've assigned a team label to this issue

Severity

annoying

Version

2022.3. probably any.

Latest Version

I could reproduce the problem in the latest build

What happened?

Deployment occasionally times out when retention is running at the same time. Specifically it happens when the deployment is inserting into DeploymentRelatedMachine. This seems related to https://github.com/OctopusDeploy/Issues/issues/7435

Reproduction

See thread and ticket for details.

Error and Stacktrace

                    |   2022-11-18T12:27:22 INSERT INTO [dbo].[DeploymentRelatedMachine]  ([DeploymentId], [MachineId])  VALUES
                    |   (@0__DeploymentId, @0__MachineId)
                    |   ,(@1__DeploymentId, @1__MachineId)
                    |   ,(@2__DeploymentId, @2__MachineId)
                    |   ,(@3__DeploymentId, @3__MachineId)
                    |   ,(@4__DeploymentId, @4__MachineId)
                    |   Transaction 'SpacesCache' Open with 0 commands started at 2022-11-18T12:28:22 (0.02 seconds ago)
                    |   System.Exception
                    |   at Nevermore.CommandExecutor.WrapException(Exception ex)
                    |   at Nevermore.CommandExecutor.ExecuteNonQueryAsync(CancellationToken cancellationToken)
                    |   at Nevermore.Advanced.ReadTransaction.ExecuteNonQueryAsync(PreparedCommand preparedCommand, CancellationToken cancellationToken)
                    |   at Nevermore.Advanced.WriteTransaction.ExecuteDataModificationAsync(PreparedCommand command, CancellationToken cancellationToken)
                    |   at Nevermore.Advanced.WriteTransaction.InsertManyAsync[TDocument](IReadOnlyCollection`1 documents, InsertOptions options, CancellationToken cancellationToken)
                    |   at Octopus.Core.RelationalStorage.RelatedDocumentStore.PopulateDeploymentRelatedMachines(IWriteQueryExecutor transaction, Deployment deployment, CancellationToken cancellationToken) in RelatedDocumentStore.cs:line 101
                    |   at Octopus.Core.RelationalStorage.RelatedDocumentStore.PopulateRelatedDocumentsAsync[TDocument](IWriteTransaction transaction, TDocument instance, CancellationToken cancellationToken) in RelatedDocumentStore.cs:line 49
                    |   at Nevermore.Advanced.WriteTransaction.UpdateAsync[TDocument](TDocument document, UpdateOptions options, CancellationToken cancellationToken)
                    |   at Octopus.Core.RelationalStorage.RawRelationalTransaction.Update[TKey](IId`1 document, CancellationToken cancellationToken, UpdateOptions options) in RawRelationalTransaction.cs:line 162
                    |   at Octopus.Server.Web.Infrastructure.OctopusQueryExecutor.Update[TKey](IId`1 instance, CancellationToken cancellationToken, UpdateOptions options) in OctopusQueryExecutor.cs:line 333
                    |   at Octopus.Core.Persistence.Database.ProjectPathDecorators.ProjectPathQueryExecutorDecorator.Update[TKey](IId`1 document, CancellationToken cancellationToken, UpdateOptions options) in ProjectPathQueryExecutorDecorator.cs:line 159
                    |   at Octopus.Core.Persistence.Database.SlugDecorators.SlugQueryExecutorDecorator.Update[TKey](IId`1 document, CancellationToken cancellationToken, UpdateOptions options) in SlugQueryExecutorDecorator.cs:line 155
                    |   at Octopus.Server.Orchestration.ServerTasks.Deploy.DeploymentPlanService.SaveStateContainer(Deployment execution, CancellationToken cancellationToken) in DeploymentPlanService.cs:line 112
                    |   at Octopus.Server.Orchestration.ServerTasks.Deploy.ExecutionPlanService`3.Cleanup(TExecutionPlan plan, CancellationToken cancellationToken) in ExecutionPlanService.cs:line 126
                    |   at Octopus.Server.Orchestration.ServerTasks.Deploy.DeploymentTaskController.<>c__DisplayClass6_0.<CleanupPlan in DeploymentTaskController.cs:line 68
                    |   at Octopus.Server.Infrastructure.Orchestration.UnitsOfWork.UnitOfWorkExecutor.<>c__DisplayClass3_0`1.<Execute in UnitOfWorkExecutor.cs:line 87
                    |   at Octopus.Core.Infrastructure.UnitsOfWork.UnitOfWorkExtensionMethods.Do(IUnitOfWork unitOfWork, Func`1 action, CancellationToken cancellationToken, String name) in UnitOfWorkExtensionMethods.cs:line 93
                    |   at Octopus.Core.Infrastructure.UnitsOfWork.UnitOfWorkExtensionMethods.Do(IUnitOfWork unitOfWork, Func`1 action, CancellationToken cancellationToken, String name) in UnitOfWorkExtensionMethods.cs:line 93
                    |   at Octopus.Server.Infrastructure.Orchestration.UnitsOfWork.UnitOfWorkExecutor.Execute[T](Func`3 action, CancellationToken cancellationToken, String name) in UnitOfWorkExecutor.cs:line 90
                    |   at Octopus.Server.Orchestration.ServerTasks.Deploy.DeploymentTaskController.CleanupPlan(DeploymentPlan plan, CancellationToken cancellationToken) in DeploymentTaskController.cs:line 70
                    |   at Octopus.Server.Orchestration.ServerTasks.Deploy.ExecutionTaskController`3.ExecuteBase(CancellationToken cancellationToken) in ExecutionTaskController.cs:line 121
                    |   at Octopus.Server.Orchestration.ServerTasks.Deploy.DeploymentTaskController.Execute(CancellationToken cancellationToken) in DeploymentTaskController.cs:line 60
                    |   at Octopus.Server.Orchestration.ServerTasks.RunningTask.<>c__DisplayClass31_0.<WorkerTask in RunningTask.cs:line 175
                    |   at Octopus.Core.Infrastructure.UnitsOfWork.UnitOfWorkExtensionMethods.Do(IUnitOfWork unitOfWork, Func`1 action, CancellationToken cancellationToken, String name) in UnitOfWorkExtensionMethods.cs:line 93
                    |   at Octopus.Core.Infrastructure.UnitsOfWork.UnitOfWorkExtensionMethods.Do(IUnitOfWork unitOfWork, Func`1 action, CancellationToken cancellationToken, String name) in UnitOfWorkExtensionMethods.cs:line 93
                    |   at Octopus.Server.Orchestration.ServerTasks.RunningTask.WorkerTask(CancellationToken cancellationToken) in RunningTask.cs:line 208
                    |   --Inner Exception--
                    |   SQL Error -2 - Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.
                    |   Microsoft.Data.SqlClient.SqlException
                    |   at Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
                    |   at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
                    |   at Microsoft.Data.SqlClient.SqlCommand.InternalEndExecuteNonQuery(IAsyncResult asyncResult, Boolean isInternal, String endMethod)
                    |   at Microsoft.Data.SqlClient.SqlCommand.EndExecuteNonQueryInternal(IAsyncResult asyncResult)
                    |   at Microsoft.Data.SqlClient.SqlCommand.EndExecuteNonQueryAsync(IAsyncResult asyncResult)
                    |   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
                    |   at Nevermore.Transient.DbCommandExtensions.<>c__DisplayClass1_0.<ExecuteNonQueryWithRetryAsync
                    |   at Nevermore.Transient.DbCommandExtensions.<>c__DisplayClass1_0.<ExecuteNonQueryWithRetryAsync
                    |   at Nevermore.CommandExecutor.ExecuteNonQueryAsync(CancellationToken cancellationToken)
                    |   --Inner Exception--
                    |   The wait operation timed out.
                    |   System.ComponentModel.Win32Exception

More Information

slack zendesk shortcut

Workaround

As it only occasionally happens, a rerun should fix it most of the times