HTTP 500 errors - "The task was cancelled" during UI actions and creating releases from build server

[david-staniec-octopus]

Team

• [X] I've assigned a team label to this issue

Severity

No response

Version

Identified in 2022.3.10437

Latest Version

No response

What happened?

Intermittent 500 errors when performing UI actions and when creating releases from a build server.

Reproduction

Reproduction steps not yet identified.

Error and Stacktrace

Example stacktraces:

An exception was thrown while trying to establish a principal for the current request
System.OperationCanceledException: The operation was canceled.
   at System.Threading.CancellationToken.ThrowOperationCanceledException()
   at System.Linq.AsyncIteratorBase`1.GetAsyncEnumerator(CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 33
   at System.Runtime.CompilerServices.ConfiguredCancelableAsyncEnumerable`1.GetAsyncEnumerator()
   at System.Threading.Tasks.AsyncEnumerableExt.GetConfiguredAsyncEnumerator[T](IAsyncEnumerable`1 enumerable, CancellationToken cancellationToken, Boolean continueOnCapturedContext) in /_/Ix.NET/Source/System.Linq.Async/System/Threading/Tasks/AsyncEnumerableExt.cs:line 14
   at System.Linq.AsyncEnumerable.TryGetFirst[TSource](IAsyncEnumerable`1 source, Func`2 predicate, CancellationToken cancellationToken)
   at System.Linq.AsyncEnumerable.<FirstOrDefaultAsync>g__Core|92_0[TSource](IAsyncEnumerable`1 source, Func`2 predicate, CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/FirstOrDefault.cs:line 59
   at Octopus.Server.Web.Infrastructure.Authentication.UserAuthenticator.AuthenticateRequest(HttpContext context, CancellationToken cancellationToken) in C:\Development\OctopusDeploy\source\Octopus.Server\Web\Infrastructure\Authentication\UserAuthenticator.cs:line 51

An exception was thrown while trying to establish a principal for the current request
System.Exception: Error while executing SQL command in transaction 'UserQuerier.TryGetByApiKey': The request failed to run because the batch is aborted, this can be caused by abort signal sent from client, or another request is running in the same session, which makes the session busy.
Operation cancelled by user.
The command being executed was:
SELECT TOP 1 Id,Username,IsActive,IsService,IdentificationToken,EmailAddress,ExternalId,ExternalIdentifiers,DisplayName,DataVersion,Created,JSON FROM [dbo].[User] WHERE [Id] = @Id
 ---> Microsoft.Data.SqlClient.SqlException (0x80131904): The request failed to run because the batch is aborted, this can be caused by abort signal sent from client, or another request is running in the same session, which makes the session busy.
Operation cancelled by user.
   at Microsoft.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__207_0(Task`1 result)
   at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
   at Nevermore.Transient.DbCommandExtensions.<>c__DisplayClass3_0.<<ExecuteReaderWithRetryAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Nevermore.Transient.DbCommandExtensions.<>c__DisplayClass3_0.<<ExecuteReaderWithRetryAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Nevermore.Transient.RetryPolicy.ExecuteActionAsync[TResult](Func`1 func)
   at Nevermore.Transient.DbCommandExtensions.ExecuteReaderWithRetryAsync(DbCommand command, RetryPolicy commandRetryPolicy, CommandBehavior commandBehavior, CancellationToken cancellationToken, RetryPolicy connectionRetryPolicy, String operationName)
   at Nevermore.CommandExecutor.ExecuteReaderAsync(CancellationToken cancellationToken)

More Information

No response

Workaround

No response

[Clare-Octopus]

Ticket 1 (Internal) - https://octopus.zendesk.com/agent/tickets/97523 Ticket 2 (Internal) - https://octopus.zendesk.com/agent/tickets/94202