Question: Parallel execution of scenarios

[marc-mueller]

SpecFlow Version:

• [X] 3.1
• [ ] 3.0
• [ ] 2.4
• [ ] 2.3
• [ ] 2.2
• [ ] 2.1
• [ ] 2.0
• [ ] 1.9

Used Test Runner

• [ ] SpecFlow+Runner
• [X] MSTest
• [X] NUnit
• [ ] Xunit

Version number:

Project Format of the SpecFlow project

• [ ] Classic project format using packages.config
• [X] Classic project format using <PackageReference> tags
• [ ] Sdk-style project format

.feature.cs files are generated using

• [X] SpecFlow.Tools.MsBuild.Generation NuGet package
• [ ] SpecFlowSingleFileGenerator custom tool

Visual Studio Version

• [X] VS 2019
• [ ] VS 2017
• [ ] VS 2015

Enable SpecFlowSingleFileGenerator Custom Tool option in Visual Studio extension settings

• [ ] Enabled
• [ ] Disabled

Are the latest Visual Studio updates installed?

• [X] Yes
• [ ] No, I use Visual Studio version <Major>.<Minor>.<Patch>

.NET Framework:

• [ ] >= .NET 4.5
• [ ] before .NET 4.5
• [ ] .NET Core 2.0
• [ ] .NET Core 2.1
• [ ] .NET Core 2.2
• [X] .NET Core 3.0

Test Execution Method:

• [X] Visual Studio Test Explorer
• [ ] TFS/VSTS/Azure DevOps – Task – PLEASE SPECIFY THE NAME OF THE TASK
• [X] Command line – VSTest.Cosole.exe

<SpecFlow> Section in app.config or content of specflow.json

Repro Project

• none

Issue Description

We currently have a large project using SpecFlow for functional testing on a system level. We now have a new requirement where we need to put the system under load. The current concept is to use the existing test case and execute them many times over a defined period of time (i.e. 50 test cases, execute a test case every 10 seconds, over a runtime of 2 hours). For parts not using Specflow, we wrote a custom Attribute which executes the given test method multiple times according to a load test configuration. If we do the same thing for SpecFlow based tests, SpecFlow is crashing. (We addes a custom MSBuild target to replace the standard attributes with our custom ones). As soon as we run a test method in parallel in another Thread, the erros happen. If we use our attributes without multithreading (sequentially executing the same test method multiple times with the same thead), it is working fine.

The easiest way to reproduce this behavior is to add two [TestCase] attributes and the [Parallelize] attribute to a NUnit generated test method.

If we run the same method multiple times in parallel, we get this error in the first execution:

Message: 
        System.ArgumentException : An item with the same key has already been added. Key: TechTalk.SpecFlow.Plugins.RuntimePluginEvents
      Stack Trace: 
        Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
        Dictionary`2.Add(TKey key, TValue value)
        ObjectContainer.Resolve(Type typeToResolve, ResolutionList resolutionPath, String name)
        ObjectContainer.Resolve(Type typeToResolve, String name)
        ObjectContainer.Resolve[T](String name)
        ObjectContainer.Resolve[T]()
        ContainerBuilder.CreateScenarioContainer(IObjectContainer testThreadContainer, ScenarioInfo scenarioInfo)
        ContextManager.InitializeScenarioContext(ScenarioInfo scenarioInfo)
        TestExecutionEngine.OnScenarioInitialize(ScenarioInfo scenarioInfo)
        TestRunner.OnScenarioInitialize(ScenarioInfo scenarioInfo)
        SpecFlowFeature1Feature.ScenarioInitialize(ScenarioInfo scenarioInfo)
        SpecFlowFeature1Feature.AddTwoNumbers() line 7

and the second execution provokes this error:

Message: 
        System.ArgumentNullException : Value cannot be null. (Parameter 'ScenarioContext')
        TearDown : System.NullReferenceException : Object reference not set to an instance of an object.
      Stack Trace: 
        TestExecutionEngine.HandleBlockSwitch(ScenarioBlock block)
        TestExecutionEngine.ExecuteStep(IContextManager contextManager, StepInstance stepInstance)
        TestExecutionEngine.Step(StepDefinitionKeyword stepDefinitionKeyword, String keyword, String text, String multilineTextArg, Table tableArg)
        TestRunner.And(String text, String multilineTextArg, Table tableArg, String keyword)
        SpecFlowFeature1Feature.AddTwoNumbers() line 9
        --TearDown
        TestExecutionEngine.OnScenarioEnd()
        TestRunner.OnScenarioEnd()
        SpecFlowFeature1Feature.TestTearDown()

Is there any way to get the tests executed in parallel? Thanks for your help and feedbasck!

Best Marc

Steps to Reproduce

Test Assembly:

[assembly: LevelOfParallelism(100)]
[assembly: Parallelizable(ParallelScope.Fixtures)]

Rewrite generated test case:

[NUnit.Framework.TestCase()]
[NUnit.Framework.TestCase()]
[NUnit.Framework.Parallelizable(NUnit.Framework.ParallelScope.All)]
[NUnit.Framework.DescriptionAttribute("Add two numbers")]
public virtual void AddTwoNumbers()
{}
```

[SabotageAndi]

@marc-mueller Sorry that we oversaw this issue. Did you find an answer to the question?

[EjaYF]

I ran into a simular issue today, however my problem was related to multiple asynchronous access to the Dictionary of a single ScenarioContext... Maybe this is related to your problem.

I think this is because the Dictionary behind the ScenarioContext is not thread-safe (the SpecFlowContext is inherited from Dictionary and not from ConcurrentDictionary).

A work-around would be to use your own ConcurrentDictionary inside a single ScenarioContext keyvalue pair to store these values or use another kind of semaphore for this purpose.

I was trying to create a pull request, but at first I have to downgrade from .NET 6 to .NET 5 on my development machine in order te create one...