SpecFlowOSS
Specflow+ runner fails with unhandled exception when running in azure devops pipeline
SpecFlow Version
3.9.8
Which test runner are you using?
SpecFlow+ Runner
Test Runner Version Number
3.9.7
.NET Implementation
.NET 5.0
Project Format of the SpecFlow project
Classic project format using packages.config
.feature.cs files are generated using
SpecFlow.Tools.MsBuild.Generation NuGet package
Test Execution Method
TFS/VSTS/Azure DevOps – Task – PLEASE SPECIFY THE NAME OF THE TASK
SpecFlow Section in app.config or content of specflow.json
{
"bindingCulture": {
"language": "en-us"
},
"language": {
"feature": "en-us"
},
"stepAssemblies": [
{
"assembly": "test-common"
}
]
}
Test-common contains some common test steps from an nuget package in our on own azure artefactory
Issue Description
When switching my existing .net 5.0 test project from NUnit to Specflow+ runner in Azure pipeline execution during dotnet test step
2021-11-10T03:06:48.1408600+00:00:TargetFramework: .NETCoreApp,Version=v5.0
2021-11-10T03:06:48.1411543+00:00:Tfm: netcoreapp5.0
2021-11-10T03:06:48.1411645+00:00:[AssemblyCache] Trying to resolve 'TechTalk.SpecRun.Framework.Executor' in folder 'D:\a\1\a\ae-gui-test\SpecFlowPlusRunner\netcoreapp5.0'
2021-11-10T03:06:48.1411790+00:00:[AssemblyCache] Trying to resolve 'TechTalk.SpecRun.Framework.Executor' to 'D:\a\1\a\ae-gui-test\SpecFlowPlusRunner\netcoreapp5.0\TechTalk.SpecRun.Framework.Executor.dll'
2021-11-10T03:06:48.1412344+00:00:[AssemblyCache] Trying to resolve 'TechTalk.SpecRun.Framework.Executor' to 'D:\a\1\a\ae-gui-test\SpecFlowPlusRunner\netcoreapp5.0\TechTalk.SpecRun.Framework.Executor.exe'
2021-11-10T03:06:48.1412763+00:00:[AssemblyCache] Could not resolve 'TechTalk.SpecRun.Framework.Executor' in folder 'D:\a\1\a\ae-gui-test\SpecFlowPlusRunner\netcoreapp5.0'
2021-11-10T03:06:48.1612894+00:00:Thread #0: Error while unpacking executor: System.ArgumentException: Path "TechTalk.SpecRun.Framework.dll" is not an absolute path. (Parameter 'assemblyPath')
at System.Runtime.Loader.AssemblyLoadContext.LoadFromAssemblyPath(String assemblyPath)
at TechTalk.SpecRun.Framework.FrameworkDependent.Remoting.AssemblyLoadContextReference.CreateObjectOfTypeFromAssembly(String assemblyName, String typeFullName)
at TechTalk.SpecRun.Framework.FrameworkDependent.Remoting.AssemblyLoadContextReference.CreateObjectOfType[T](String assemblyName, String typeFullName)
at TechTalk.SpecRun.Framework.Execution.SharedAppDomain.SharedAppDomainTestAssemblyExecutorHost.Initialize(Int32 threadId, ITestExecutionManager executionManager, IAssemblyReference testAssembly, ITestLogger logger, String testAssemblyFullPath, String testAssemblyConfigFilePath, TestExecutionConfiguration testExecutionConfiguration, String target)
at TechTalk.SpecRun.Framework.TestThread.InitializeExecutor(ITestAssemblyExecutor executor, DeploymentContext deploymentContext, IAssemblyReference testAssembly)
2021-11-10T03:06:48.1631614+00:00:test thread #0 error: System.ArgumentException: Path "TechTalk.SpecRun.Framework.dll" is not an absolute path. (Parameter 'assemblyPath')
at TechTalk.SpecRun.Framework.TestThread.UnpackTestThreadExecutorInfoResult(IResult`1 testThreadExecutorInfoResult)
at TechTalk.SpecRun.Framework.TestThread.InitializeTestThreadExecutor(IAssemblyReference testAssembly, ExecutionModelSettings executionModelSettings, String testTarget)
at TechTalk.SpecRun.Framework.TestThread.GetThreadExecutor(IAssemblyReference testAssembly, ExecutionModelSettings executionModelSettings, String testTarget)
at TechTalk.SpecRun.Framework.TestThread.GetThreadExecutorForTestItem(TestItem testItem)
at TechTalk.SpecRun.Framework.TestThread.RunNonEmptyTestSuite(CancellationToken cancellationToken, TestThreadState testThreadState)
at TechTalk.SpecRun.Framework.TestThread.Run(ITestExecutionManager executionManagerForRun, CancellationToken cancellationToken)
at TechTalk.SpecRun.Framework.AsyncTestThreadRunner.RunSync(TestExecutionManager executionManager, CancellationToken cancellationToken)
2021-11-10T03:06:48.1659213+00:00:test thread error: TechTalk.SpecRun.Framework.SpecRunException: At least one test thread aborted.
All test scenarios are finally skipped due to this exception. One thing I notice is that the target framework shows .NetCoreApp 5.0, instead of expected .Net 5.0
Steps to Reproduce
The change includes
- Update specflow to 3.9.8 from 3.8.14
- Add reference to SpecRun.Runner (3.9.7) and SpecRun.SpecFlow (3.9.7)
- Remove reference to NUnit3TestAdapter and SpecFlow.NUnit
- Created Default.srprofile as below
<?xml version="1.0" encoding="utf-8"?>
<TestProfile xmlns="http://www.specflow.org/schemas/plus/TestProfile/1.5">
<Settings projectName="Namespace.GUITest" />
<Execution stopAfterFailures="100" testThreadCount="1" testSchedulingMode="Sequential" retryCount="1" retryFor="Failing" apartmentState="STA"/>
<!-- For collecting by a SpecRun server update and enable the following element. For using the
collected statistics, set testSchedulingMode="Adaptive" attribute on the <Execution> element.
<Server serverUrl="http://specrunserver:6365" publishResults="true" />
-->
<Environment testThreadIsolation="SharedAppDomain" framework="Net5.0"/>
<TestAssemblyPaths>
<TestAssemblyPath>Namespace.GUITest.dll</TestAssemblyPath>
</TestAssemblyPaths>
<Report copyAlsoToBaseFolder="true"/>
<DeploymentTransformation>
<Steps>
<!-- sample config transform to change the connection string-->
<!--<ConfigFileTransformation configFile="App.config">
<Transformation>
<![CDATA[<?xml version="1.0" encoding="utf-8"?>
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
<connectionStrings>
<add name="MyDatabase" connectionString="Data Source=.;Initial Catalog=MyDatabaseForTesting;Integrated Security=True"
xdt:Locator="Match(name)" xdt:Transform="SetAttributes(connectionString)" />
</connectionStrings>
</configuration>
]]>
</Transformation>
</ConfigFileTransformation>-->
</Steps>
</DeploymentTransformation>
</TestProfile>
The failing dotnet test step in Azure Devops yaml is
- task: DotNetCoreCLI@2
displayName: "Tests Execution - ${{parameters.stageName}}"
inputs:
command: test
projects: '$(build.artifactstagingdirectory)/${{parameters.projectName}}/**/${{parameters.projectName}}.dll'
publishTestResults: true
modifyOutputPath: true
arguments: --filter "((TestCategory=${{parameters.testFilters}})&TestCategory!=off)"
All test scenarios are always skipped, with exception as shown in description. The whole log file is attached MottMac.AnalyticalEngine.GUITest_MottMac.AnalyticalEngine.GUITest_2021-11-10T030647.log
Running the same dotnet cli command in local (by extracting the command line displayed in azure pipeline console) is all fine, with only numerous of warning that assembly Cache file are not found (likely to be due to TFM is NetCoreApp, instead of .Net)
Link to Repro Project
No response
A new finding I have is that the problem no longer exists in azure pipeline if I switch my test project into .net core 3.1
Strange issue. Do you make a clean build on the build agents? Perhaps there are some leftovers with old versions that make problems.
Yes both agents and build are clean.
The agents are claimed only from Microsoft as we don't host any in Azure pipeline. So they are completely clean without any previous code on the test project
FYI on detail of components in their agent - https://github.com/actions/virtual-environments/blob/main/images/win/Windows2019-Readme.md
It worked fine using NUnit. We decided to switch to Specflow+ runner as NUnit + Specflow can no longer support retry on fail
Ah, the retry functionality. Did you tried the one included in the VS Test task?
