Random NullReferenceException when run Specflow.xUnit tests in parallel

[ttranbunnings]

SpecFlow Version

3.9.40

Which test runner are you using?

xUnit

Test Runner Version Number

xunit.runner.visualstudio/2.4.3

.NET Implementation

.NET 5.0

Project Format of the SpecFlow project

Sdk-style project format

.feature.cs files are generated using

SpecFlowSingleFileGenerator custom tool

Test Execution Method

Command line – PLEASE SPECIFY THE FULL COMMAND LINE

SpecFlow Section in app.config or content of specflow.json

No response

Issue Description

Tests randomly failed when run in parallel. Works fine when run in Debug or disable parallel[assembly: Xunit.CollectionBehavior(DisableTestParallelization = true)] System.NullReferenceException Object reference not set to an instance of an object. at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.OnScenarioEnd() at TechTalk.SpecFlow.TestRunner.OnScenarioEnd() at BayAge.Test.Features.BayEntry.BayQueryFeature.TestTearDown() at BayAge.Test.Features.BayEntry.BayQueryFeature.System.IDisposable.Dispose() at Xunit.Sdk.ExecutionTimer.Aggregate(Action action) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\ExecutionTimer.cs:line 31 at ReflectionAbstractionExtensions.DisposeTestClass(ITest test, Object testClass, IMessageBus messageBus, ExecutionTimer timer, CancellationTokenSource cancellationTokenSource) in C:\Dev\xunit\xunit\src\xunit.execution\Extensions\ReflectionAbstractionExtensions.cs:line 79 at Xunit.Sdk.TestInvoker1.<>c__DisplayClass47_0.<RunAsync>b__1() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestInvoker.cs:line 216 at Xunit.Sdk.ExceptionAggregator.Run(Action code) in C:\Dev\xunit\xunit\src\xunit.core\Sdk\ExceptionAggregator.cs:line 73 `

` System.AggregateException: One or more errors occurred. (Object reference not set to an instance of an object.) (Object reference not set to an instan...

System.AggregateException One or more errors occurred. (Object reference not set to an instance of an object.) (Object reference not set to an instance of an object.) Exception doesn't have a stacktrace

System.NullReferenceException Object reference not set to an instance of an object. at TechTalk.SpecFlow.Infrastructure.ContextManager.InitializeScenarioContext(ScenarioInfo scenarioInfo) at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.OnScenarioInitialize(ScenarioInfo scenarioInfo) at TechTalk.SpecFlow.TestRunner.OnScenarioInitialize(ScenarioInfo scenarioInfo)`

Steps to Reproduce

Run dotnet test in terminal or using Jetbrain Rider run tests

Link to Repro Project

No response

[SabotageAndi]

Please provide a project where we can reproduce this issue. Thanks.

[avisra]

I am running into the same problem - when I have multiple scenarios in a single feature.

[szarykott]

As a quick workaround, I used NUnit runner that runs the same tests just fine. Not a perfect solution, but it works.

And a side note - it would still fail if code was built in Debug mode for me. Running tests sequentially was not an option, so I did not check if this fixed the issue.

[VGib]

same problem here.

SpecFlow: 3.9.58 SpecFlow.XUnit: 3.9.58 XUnit: 2.4.1

on Visual Studio 2022 and azure devops build

[SabotageAndi]

I still need a project to reproduce the issue to be able to do anything.

[VGib]

Hello,

i'm using a client projet, so i can't share the projet. The project uses Azure and database objects which needs to be disposed. I will try to reproduce the problem.

Regards,

VGib

[burciugi]

same problem here.

SpecFlow: 3.9.58 SpecFlow.XUnit: 3.9.58 XUnit: 2.4.1

on Visual Studio 2022 and azure devops build

Hello @VGib, I'm getting the same issues when trying to run feature files with multiple scenarios, I'm using the same stack. Did you manage to find a workaround or fix?

Thanks, Gabriel

[304NotModified]

Same issue here:

SpecFlow: 3.9.69 SpecFlow.XUnit: 3.9.69 SpecFlow.LivingDoc: 3.9.57 XUnit: 2.4.1 AdCodicem.SpecFlow.MicrosoftDependencyInjection: 1.1.8

Stacktrace:

System.AggregateException : One or more errors occurred. (Invalid argument) (Object reference not set to an instance of an object.)
---- System.IO.IOException : Invalid argument
---- System.NullReferenceException : Object reference not set to an instance of an object.

----- Inner Stack Trace #1 (System.IO.IOException) -----
   at System.IO.Enumeration.FileSystemEnumerator`1.FindNextEntry(Byte* entryBufferPtr, Int32 bufferLength)
   at System.IO.Enumeration.FileSystemEnumerator`1.MoveNext()
   at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
   at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
   at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
   at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
   at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
   at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
   at System.IO.FileSystemWatcher.RunningInstance..ctor(FileSystemWatcher watcher, SafeFileHandle inotifyHandle, String directoryPath, Boolean includeSubdirectories, NotifyFilters notifyFilters, CancellationToken cancellationToken)
   at System.IO.FileSystemWatcher.StartRaisingEvents()
   at System.IO.FileSystemWatcher.StartRaisingEventsIfNotDisposed()
   at System.IO.FileSystemWatcher.set_EnableRaisingEvents(Boolean value)
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher()
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(String filter)
   at Microsoft.Extensions.FileProviders.PhysicalFileProvider.Watch(String filter)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider.<.ctor>b__1_0()
   at Microsoft.Extensions.Primitives.ChangeToken.OnChange(Func`1 changeTokenProducer, Action changeTokenConsumer)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider..ctor(FileConfigurationSource source)
   at Microsoft.Extensions.Configuration.Json.JsonConfigurationSource.Build(IConfigurationBuilder builder)
   at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
   at MyTests.DependencyContainerInitializer.CreateConfiguration() 
   at MyTests.DependencyContainerInitializer.ConfigureServices(IServiceCollection services)
   at AdCodicem.SpecFlow.MicrosoftDependencyInjection.DependencyInjectionPlugin.ConfigureServices(IObjectContainer delegateContainer)
   at AdCodicem.SpecFlow.MicrosoftDependencyInjection.DependencyInjectionPlugin.<Initialize>b__3_1(Object _, CustomizeScenarioDependenciesEventArgs args)
   at TechTalk.SpecFlow.Plugins.RuntimePluginEvents.RaiseCustomizeScenarioDependencies(ObjectContainer scenarioContainer)
   at TechTalk.SpecFlow.Infrastructure.ContainerBuilder.CreateScenarioContainer(IObjectContainer testThreadContainer, ScenarioInfo scenarioInfo)
   at TechTalk.SpecFlow.Infrastructure.ContextManager.InitializeScenarioContext(ScenarioInfo scenarioInfo)
   at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.OnScenarioInitialize(ScenarioInfo scenarioInfo)
   at TechTalk.SpecFlow.TestRunner.OnScenarioInitialize(ScenarioInfo scenarioInfo)
   at MyTests.Feature1.ScenarioInitialize(ScenarioInfo scenarioInfo)
   at MyTests.Feature1.Test1(String url, String[] exampleTags)
----- Inner Stack Trace #2 (System.NullReferenceException) -----
   at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.OnScenarioEnd()
   at TechTalk.SpecFlow.TestRunner.OnScenarioEnd()
   at MyTests.Feature1.TestTearDown()
   at MyTests.Feature1.System.IDisposable.Dispose()
   at ReflectionAbstractionExtensions.DisposeTestClass(ITest test, Object testClass, IMessageBus messageBus, ExecutionTimer timer, CancellationTokenSource cancellationTokenSource)

[burciugi]

@304NotModified We got rid of this issue by switching to NUnit.

[304NotModified]

I still need a project to reproduce the issue to be able to do anything.

I guess this is hard to ask. It's too unstable for that.

Isn't the stacktrace clear?

@304NotModified We got rid of this issue by switching to NUnit.

Thanks. That isn't an option for us (policies), but [assembly: Xunit.CollectionBehavior(DisableTestParallelization = true)] is fine to us.

[burciugi]

@304NotModified If you don't care about running tests in parallel that's fine, but we need to run them in parallel so that's why we switched to NUnit.

[dinaRodeny]

same issue