Unawaited tasks in Amplitude telemetry cause `MultipleActiveResultSets` exception

[droyad]

Severity

Prevents collection of Telemetry, contributes to connection leak

Version

2022.2.2228

Latest Version

I could reproduce the problem in the latest build

What happened?

When an exception occurs in this Select's lambda before the first async work (not necessarily the first await), any previously created tasks from that lambda are not awaited.

The exception that triggers this is highly likely #8839

The exception triggered from this Issue causes #8841 to leak connections

Reproduction

Getting these exceptions does depend on timing, the robw/reproduce-8840 branch sets up this reasonably reliably. To trigger it, run a deployment with at least one target.

Error and Stacktrace

The unawaited tasks may get the following exception if they need to resolves something from the container


Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed

Or an exception related to the connection being closed.

It will likely also have the following exception

A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread.

The main thread may get the following exception if one of the unawaited tasks has an open query:

System.InvalidOperationException: The connection does not support MultipleActiveResultSets.
   at Microsoft.Data.SqlClient.SqlInternalTransaction.Rollback()
   at Octopus.Core.RelationalStorage.RawRelationalTransaction.Dispose() in ./source/Octopus.Core/RelationalStorage/RawRelationalTransaction.cs:line 153
   at Autofac.Core.Disposer.DisposeAsync(Boolean disposing)
   at Autofac.Core.Lifetime.LifetimeScope.DisposeAsync(Boolean disposing)
   at Octopus.Core.Infrastructure.MessageBus.PipelineFactory.BuildAndRun[TEvent](Int64 sequenceNumber, TEvent e, IDictionary`2 metadata, String consumerGroupId, Type handlerType, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/MessageBus/PipelineFactory.cs:line 53
   at Octopus.Core.Infrastructure.MessageBus.MessagePump.Dispatch[TEvent](Int64 sequenceNumber, TEvent e, IDictionary`2 metadata, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/MessageBus/MessagePump.cs:line 101

Workaround

None available