SpecFlow icon indicating copy to clipboard operation
SpecFlow copied to clipboard
verified publisher icon SpecFlowOSS

SpecFlow intermittently fails to load xunit plugin

Open jjmanton opened this issue 4 years ago • 1 comments

SpecFlow Version:

  • [x] 3.7
  • [ ] 3.6
  • [ ] 3.5
  • [ ] 3.4
  • [ ] 3.3
  • [ ] 3.1
  • [ ] 3.0
  • [ ] 2.4
  • [ ] 2.3
  • [ ] 2.2
  • [ ] 2.1
  • [ ] 2.0
  • [ ] 1.9

Used Test Runner

  • [ ] SpecFlow+Runner
  • [ ] MSTest
  • [ ] NUnit
  • [x] Xunit

Version number: We use a forked version of 2.4.1 that has inclusion file support here (https://github.com/xunit/xunit/pull/2018)

Project Format of the SpecFlow project

  • [ ] Classic project format using packages.config
  • [x] Classic project format using <PackageReference> tags
  • [ ] Sdk-style project format

.feature.cs files are generated using

  • [x] SpecFlow.Tools.MsBuild.Generation NuGet package
  • [ ] SpecFlowSingleFileGenerator custom tool

Visual Studio Version

  • [x] VS 2019
  • [ ] VS 2017

Enable SpecFlowSingleFileGenerator Custom Tool option in Visual Studio extension settings

  • [ ] Enabled
  • [x] Disabled

Are the latest Visual Studio updates installed?

  • [x] Yes
  • [ ] No, I use Visual Studio version <Major>.<Minor>.<Patch>

.NET Framework:

  • [x] >= .NET 4.5. (4.8)
  • [ ] before .NET 4.5
  • [ ] .NET Core 2.1
  • [ ] .NET Core 3.1
  • [ ] .NET 5.0

Test Execution Method:

  • [ ] Visual Studio Test Explorer
  • [ ] TFS/VSTS/Azure DevOps – Task – PLEASE SPECIFY THE NAME OF THE TASK
  • [x] Command line – "xunit.console.exe C:\Installs\IntegrationTests\AW.Integration.UI.Console.Tests.dll C:\Installs\IntegrationTests\AW.Integration.Api.Tests.dll -inclusionfile C:\Installs\IntegrationTests\{bucket_file} -nunit C:\Installs\IntegrationTests\TestResult\{test_output_file} -verbose -diagnostics -parallel all -maxthreads 8 > C:\Installs\IntegrationTests\TestResult\{xunit_log}\"

<SpecFlow> Section in app.config or content of specflow.json

  <specFlow>
    <stepAssemblies>
      <stepAssembly assembly="AW.Integration.UI.Framework" />
      <stepAssembly assembly="AW.Integration.UI.StepDefinitions" />
      <stepAssembly assembly="AW.Integration.Api.StepDefinitions" />
      <stepAssembly assembly="AW.Integration.UI.SSP.StepDefinitions" />
      <stepAssembly assembly="AW.Integration.MockService" />
    </stepAssemblies>
  </specFlow>

AND

  <specFlow>
    <stepAssemblies>
      <stepAssembly assembly="AW.Integration.Api.StepDefinitions" />
      <stepAssembly assembly="AW.Integration.MockService" />
    </stepAssemblies>
  </specFlow>

Issue Description

We will intermittently see our integration tests fail, with the following error -

System.AggregateException : One or more errors occurred.\r\n---- BoDi.ObjectContainerException : Interface cannot be resolved: TechTalk.SpecFlow.UnitTestProvider.IUnitTestRuntimeProvider('nunit')\r\n---- BoDi.ObjectContainerException : Interface cannot be resolved: TechTalk.SpecFlow.UnitTestProvider.IUnitTestRuntimeProvider('nunit')\r\n---- The following constructor parameters did not have matching fixture data: FixtureData fixtureData, AW_Integration_UI_Console_Tests_XUnitAssemblyFixture assemblyFixture

It appears that about 10% of the time, it will fail to load the xunit plugin and default to nunit.

When everything is working, the xunit logs will look something like this -

xUnit.net Console Runner v1.0.0-AirwatchCustom (64-bit Desktop .NET 4.0.30319.42000)
  Discovering: AW.Integration.UI.Console.Tests (app domain = on [shadow copy], method display = ClassAndMethod, method display options = None)
  Discovering: AW.Integration.Api.Tests (app domain = on [shadow copy], method display = ClassAndMethod, method display options = None)
  Discovered:  AW.Integration.UI.Console.Tests (found 570 of 1749 test cases)
  Starting:    AW.Integration.UI.Console.Tests (parallel test collections = on, max threads = 8)
-> Loading plugin C:\Installs\IntegrationTests\TechTalk.SpecFlow.xUnit.SpecFlowPlugin.dll
-> Loading plugin C:\Users\admin\AppData\Local\Temp\a1d54b74-15cc-49b7-9785-79f1917352d9\a1d54b74-15cc-49b7-9785-79f1917352d9\assembly\dl3\c913b123\6c285b3a_7705d701\AW.Integration.UI.Console.Tests.dll
-> Missing [assembly:RuntimePlugin] attribute in AW.Integration.UI.Console.Tests, Version=21.4.0.0, Culture=neutral, PublicKeyToken=null. Please check https://go.specflow.org/doc-plugins for details.
-> Using app.config
  Discovered:  AW.Integration.Api.Tests (found 1829 of 4989 test cases)
  Starting:    AW.Integration.Api.Tests (parallel test collections = on, max threads = 8)
-> Loading plugin C:\Installs\IntegrationTests\TechTalk.SpecFlow.xUnit.SpecFlowPlugin.dll
-> Loading plugin C:\Users\admin\AppData\Local\Temp\59a891a3-4c07-468f-a2ad-d29208313b8a\59a891a3-4c07-468f-a2ad-d29208313b8a\assembly\dl3\3fb08caf\1e24583b_7705d701\AW.Integration.Api.Tests.dll
-> Missing [assembly:RuntimePlugin] attribute in AW.Integration.Api.Tests, Version=21.4.0.0, Culture=neutral, PublicKeyToken=null. Please check https://go.specflow.org/doc-plugins for details.
-> Using app.config
Hosting environment: Production
Content root path: C:\Installs\IntegrationTests
Now listening on: http://0.0.0.0:6002

But occasionally we see something different -

xUnit.net Console Runner v1.0.0-AirwatchCustom (64-bit Desktop .NET 4.0.30319.42000)
  Discovering: AW.Integration.UI.Console.Tests (app domain = on [shadow copy], method display = ClassAndMethod, method display options = None)
  Discovering: AW.Integration.Api.Tests (app domain = on [shadow copy], method display = ClassAndMethod, method display options = None)
  Discovered:  AW.Integration.UI.Console.Tests (found 597 of 1749 test cases)
  Starting:    AW.Integration.UI.Console.Tests (parallel test collections = on, max threads = 8)
-> Loading plugin C:\Users\admin\AppData\Local\Temp\00adbf63-1b98-425b-9439-99b750ca0015\00adbf63-1b98-425b-9439-99b750ca0015\assembly\dl3\dbcde0be\0b87573e_7705d701\AW.Integration.UI.Console.Tests.dll
-> Missing [assembly:RuntimePlugin] attribute in AW.Integration.UI.Console.Tests, Version=21.4.0.0, Culture=neutral, PublicKeyToken=null. Please check https://go.specflow.org/doc-plugins for details.
-> Loading plugin C:\Users\admin\AppData\Local\Temp\00adbf63-1b98-425b-9439-99b750ca0015\00adbf63-1b98-425b-9439-99b750ca0015\assembly\dl3\dbcde0be\0b87573e_7705d701\AW.Integration.UI.Console.Tests.dll
-> Missing [assembly:RuntimePlugin] attribute in AW.Integration.UI.Console.Tests, Version=21.4.0.0, Culture=neutral, PublicKeyToken=null. Please check https://go.specflow.org/doc-plugins for details.
-> Loading plugin C:\Users\admin\AppData\Local\Temp\00adbf63-1b98-425b-9439-99b750ca0015\00adbf63-1b98-425b-9439-99b750ca0015\assembly\dl3\dbcde0be\0b87573e_7705d701\AW.Integration.UI.Console.Tests.dll
-> Missing [assembly:RuntimePlugin] attribute in AW.Integration.UI.Console.Tests, Version=21.4.0.0, Culture=neutral, PublicKeyToken=null. Please check https://go.specflow.org/doc-plugins for details.
        ----- Inner Stack Trace #1 (BoDi.ObjectContainerException) -----
           at BoDi.ObjectContainer.TypeRegistration.<>c__DisplayClass3_0.<ResolvePerContext>b__1()
           at BoDi.ObjectContainer.RegistrationWithStrategy.ExecuteWithLock(Object lockObject, Func`1 getter, Func`1 factory, ResolutionList resolutionPath)
-> Loading plugin C:\Users\admin\AppData\Local\Temp\00adbf63-1b98-425b-9439-99b750ca0015\00adbf63-1b98-425b-9439-99b750ca0015\assembly\dl3\dbcde0be\0b87573e_7705d701\AW.Integration.UI.Console.Tests.dll
-> Missing [assembly:RuntimePlugin] attribute in AW.Integration.UI.Console.Tests, Version=21.4.0.0, Culture=neutral, PublicKeyToken=null. Please check https://go.specflow.org/doc-plugins for details.
           at BoDi.ObjectContainer.TypeRegistration.ResolvePerContext(ObjectContainer container, RegistrationKey keyToResolve, ResolutionList resolutionPath)
           at BoDi.ObjectContainer.RegistrationWithStrategy.Resolve(ObjectContainer container, RegistrationKey keyToResolve, ResolutionList resolutionPath)
           at BoDi.ObjectContainer.ResolveObject(RegistrationKey keyToResolve, ResolutionList resolutionPath)
           at BoDi.ObjectContainer.Resolve(Type typeToResolve, ResolutionList resolutionPath, String name)
-> Loading plugin C:\Users\admin\AppData\Local\Temp\00adbf63-1b98-425b-9439-99b750ca0015\00adbf63-1b98-425b-9439-99b750ca0015\assembly\dl3\dbcde0be\0b87573e_7705d701\AW.Integration.UI.Console.Tests.dll
-> Missing [assembly:RuntimePlugin] attribute in AW.Integration.UI.Console.Tests, Version=21.4.0.0, Culture=neutral, PublicKeyToken=null. Please check https://go.specflow.org/doc-plugins for details.
           at BoDi.ObjectContainer.Resolve[T](String name)
           at TechTalk.SpecFlow.Infrastructure.ContainerBuilder.CreateGlobalContainer(Assembly testAssembly, IRuntimeConfigurationProvider configurationProvider)
           at TechTalk.SpecFlow.TestRunnerManager.CreateTestRunnerManager(Assembly testAssembly, IContainerBuilder containerBuilder)
           at TechTalk.SpecFlow.TestRunnerManager.GetTestRunnerManager(Assembly testAssembly, IContainerBuilder containerBuilder, Boolean createIfMissing)
           at TechTalk.SpecFlow.TestRunnerManager.OnTestRunStart(Assembly testAssembly, IContainerBuilder containerBuilder)
        E:\agt01\COM-CN64173-BAWAP\src\AW Integration Tests\AW.Integration.UI.Console.Tests\obj\x64\Release\xUnit.AssemblyHooks.cs(20,0): at AW_Integration_UI_Console_Tests_XUnitAssemblyFixture..ctor()
-> Loading plugin C:\Users\admin\AppData\Local\Temp\00adbf63-1b98-425b-9439-99b750ca0015\00adbf63-1b98-425b-9439-99b750ca0015\assembly\dl3\dbcde0be\0b87573e_7705d701\AW.Integration.UI.Console.Tests.dll
-> Missing [assembly:RuntimePlugin] attribute in AW.Integration.UI.Console.Tests, Version=21.4.0.0, Culture=neutral, PublicKeyToken=null. Please check https://go.specflow.org/doc-plugins for details.
        ----- Inner Stack Trace #2 (BoDi.ObjectContainerException) -----
           at BoDi.ObjectContainer.TypeRegistration.<>c__DisplayClass3_0.<ResolvePerContext>b__1()
           at BoDi.ObjectContainer.RegistrationWithStrategy.ExecuteWithLock(Object lockObject, Func`1 getter, Func`1 factory, ResolutionList resolutionPath)
-> Loading plugin C:\Users\admin\AppData\Local\Temp\00adbf63-1b98-425b-9439-99b750ca0015\00adbf63-1b98-425b-9439-99b750ca0015\assembly\dl3\dbcde0be\0b87573e_7705d701\AW.Integration.UI.Console.Tests.dll
-> Missing [assembly:RuntimePlugin] attribute in AW.Integration.UI.Console.Tests, Version=21.4.0.0, Culture=neutral, PublicKeyToken=null. Please check https://go.specflow.org/doc-plugins for details.
           at BoDi.ObjectContainer.TypeRegistration.ResolvePerContext(ObjectContainer container, RegistrationKey keyToResolve, ResolutionList resolutionPath)
           at BoDi.ObjectContainer.RegistrationWithStrategy.Resolve(ObjectContainer container, RegistrationKey keyToResolve, ResolutionList resolutionPath)
           at BoDi.ObjectContainer.ResolveObject(RegistrationKey keyToResolve, ResolutionList resolutionPath)
-> Loading plugin C:\Users\admin\AppData\Local\Temp\00adbf63-1b98-425b-9439-99b750ca0015\00adbf63-1b98-425b-9439-99b750ca0015\assembly\dl3\dbcde0be\0b87573e_7705d701\AW.Integration.UI.Console.Tests.dll
-> Missing [assembly:RuntimePlugin] attribute in AW.Integration.UI.Console.Tests, Version=21.4.0.0, Culture=neutral, PublicKeyToken=null. Please check https://go.specflow.org/doc-plugins for details.
           at BoDi.ObjectContainer.Resolve(Type typeToResolve, ResolutionList resolutionPath, String name)
           at BoDi.ObjectContainer.Resolve[T](String name)
           at TechTalk.SpecFlow.Infrastructure.ContainerBuilder.CreateGlobalContainer(Assembly testAssembly, IRuntimeConfigurationProvider configurationProvider)
           at TechTalk.SpecFlow.TestRunnerManager.CreateTestRunnerManager(Assembly testAssembly, IContainerBuilder containerBuilder)
-> Loading plugin C:\Users\admin\AppData\Local\Temp\00adbf63-1b98-425b-9439-99b750ca0015\00adbf63-1b98-425b-9439-99b750ca0015\assembly\dl3\dbcde0be\0b87573e_7705d701\AW.Integration.UI.Console.Tests.dll
           at TechTalk.SpecFlow.TestRunnerManager.GetTestRunnerManager(Assembly testAssembly, IContainerBuilder containerBuilder, Boolean createIfMissing)
-> Missing [assembly:RuntimePlugin] attribute in AW.Integration.UI.Console.Tests, Version=21.4.0.0, Culture=neutral, PublicKeyToken=null. Please check https://go.specflow.org/doc-plugins for details.
           at TechTalk.SpecFlow.TestRunnerManager.GetTestRunner(Assembly testAssembly, Nullable`1 managedThreadId, IContainerBuilder containerBuilder)
           at AW.Integration.UI.Console.Tests.Features.Modules.Enrollment.Restriction.EnrollmentRestrictionPolicyAssignmentFeature.FeatureSetup()
        ----- Inner Stack Trace #3 (Xunit.Sdk.TestClassException) -----

Steps to Reproduce

I am suspecting that this has something to do with the fact that we pass two assemblies to xunit runner for execution, and there could be some threading/locking problem. We are seeing this about 10% of our runs right now on the 3.7 upgrade branch. We are NOT seeing this issue on 3.0. I believe it occurs on any version 3.1+

Some additional background, if it is helpful - we are splitting our tests to 3 parallel runners, with a random selection from our API test suite and our UI test suite. In total, there are around 7000 tests to execute.

Repro Project

jjmanton avatar Feb 18 '21 15:02 jjmanton

Sorry for the late answer. Can you deactivate the parallel execution of different assemblies? So that it executes first all scenarios in parallel from the first assembly and then executes all scenarios in parallel of the second assembly?

SabotageAndi avatar Mar 30 '21 14:03 SabotageAndi