intermittent exceptions from WorkflowLoggingService

[TomasEkeli]

Expected Behavior

It should be possible to run outside-in tests on an aspnetcore application with dapr without intermittent failures.

Actual Behavior

sometimes when i run my test suite (not always) one of many tests will fail with a stack-trace telling me that something was added to a hash-set in a way that is not allowed.

I am using dapr.aspnetcore and dapr.workflow version 1.13.0, with CLI version 1.13.0 and runtime version 1.13.2

I am using dotnet8 in a devcontainer.

i have set up my aspnetcore app with outside-in tests that use a WebApplicationFactory.

note: the failing test is not the same one between runs, and the failing tests are not related to dapr.

test-run-log with stack-trace (with references to my code expunged):

Test run for /***/bin/Debug/net8.0/***.tests.dll (.NETCoreApp,Version=v8.0)
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
[xUnit.net 00:00:01.98]     *** [FAIL]
  Failed *** [1 ms]
  Error Message:
   System.InvalidOperationException : Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
  Stack Trace:
     at System.Collections.Generic.HashSet`1.AddIfNotPresent(T value, Int32& location)
   at System.Collections.Generic.HashSet`1.Add(T item)
   at Dapr.Workflow.WorkflowLoggingService.LogWorkflowName(String workflowName)
   at Dapr.Workflow.WorkflowRuntimeOptions.<>c__DisplayClass3_0`1.<RegisterWorkflow>b__0(DurableTaskRegistry registry)
   at Dapr.Workflow.WorkflowRuntimeOptions.AddWorkflowsAndActivitiesToRegistry(DurableTaskRegistry registry)
   at Dapr.Workflow.WorkflowServiceCollectionExtensions.<>c__DisplayClass0_1.<AddDaprWorkflow>b__1(DurableTaskRegistry registry)
   at Microsoft.Extensions.Options.OptionsFactory`1.Create(String name)
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
   at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
   at System.Lazy`1.CreateValue()
   at Microsoft.Extensions.Options.OptionsCache`1.GetOrAdd[TArg](String name, Func`3 createOptions, TArg factoryArgument)
   at Microsoft.DurableTask.ServiceProviderExtensions.GetOptions[TOptions](IServiceProvider serviceProvider, String name)
   at Microsoft.DurableTask.Worker.DefaultDurableTaskWorkerBuilder.Build(IServiceProvider serviceProvider)
   at Microsoft.DurableTask.Worker.ServiceCollectionExtensions.<>c__DisplayClass1_0.<AddDurableTaskWorker>b__0(IServiceProvider sp)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitIEnumerable(IEnumerableCallSite enumerableCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.CreateServiceAccessor(ServiceIdentifier serviceIdentifier)
   at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(ServiceIdentifier serviceIdentifier, ServiceProviderEngineScope serviceProviderEngineScope)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
   at Program.<Main>$(String[] args) in /workspaces/transcription/src/web/Program.cs:line 57
   at Program.<Main>(String[] args)
   at InvokeStub_Program.<Main>(Object, Span`1)
   at System.Reflection.MethodBaseInvoker.InvokeWithOneArg(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Mvc.Testing.DeferredHostBuilder.DeferredHost.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Start(IHost host)
   at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.CreateHost(IHostBuilder builder)
   at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.ConfigureHostBuilder(IHostBuilder hostBuilder)
   at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.EnsureServer()
   at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.CreateDefaultClient(DelegatingHandler[] handlers)
   at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.CreateDefaultClient(Uri baseAddress, DelegatingHandler[] handlers)
   at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.CreateClient(WebApplicationFactoryClientOptions options)
   at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.CreateClient()
   at ***/TestWithBackend.cs:line 11
   at ***..ctor()
   at System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly, Boolean wrapExceptions)

Failed!  - Failed:     1, Passed:   127, Skipped:     0, Total:   128, Duration: 14 s - ***.tests.dll (net8.0)
dotnet watch ❌ Exited with error code 1

Steps to Reproduce the Problem

i guess: create a test-project of an aspnetcore project that uses workflows and registers some activities, and then run some tests (128 tests in my case) against that with a new application each time.

i am registering one workflow and 6 activities (mostly empty still)