"No test is available" with /logger:trx but everything works with /logger:console

[jnm2]

Please excuse the interesting PowerShell command. It's based on the command that Azure Pipelines VSTest task builds. I'm trying to track down the difference between an Azure Pipelines build that skips all tests and a build in a different project that actually runs the tests. It comes down to what looks like a VSTest console bug. I can repro locally using the open-source project I'm looking at. If you need it, I can spend the time to boil down the minimal files needed to demonstrate.

If you change /Logger:trx to /Logger:console OR you remove the /TestAdapterPath parameter, tests begin to be discovered and everything works properly.

C:\Users\Joseph\Source\Repos\PropertyChangedAnalyzers [(b740437...)]> &"C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" "PropertyChangedAnalyzers.Test\bin\Release\net472\PropertyChangedAnalyzers.Test.dll" /Logger:"trx" /TestAdapterPath:"."
Microsoft (R) Test Execution Command Line Tool Version 16.3.0-preview-20190715-02
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
No test is available in C:\Users\Joseph\Source\Repos\PropertyChangedAnalyzers\PropertyChangedAnalyzers.Test\bin\Release\net472\PropertyChangedAnalyzers.Test.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.      Test run in progress.Results File: C:\Users\Joseph\Source\Repos\PropertyChangedAnalyzers\TestResults\Joseph_TALIESIN_2019-11-06_17_11_12.trx

Diagnostic logs

verbose_diagnostic_log.txt

It looks like the diagnostic log is full of 283 repetitions of vstest.console.exe failing to load System.Runtime.dll. Why would these failures go away if you change /Logger:trx to /Logger:console or if you remove the /TestAdapterPath parameter?

TpTrace Warning: 0 : 22240, 4, 2019/11/06, 17:16:05.318, 3276125106496, vstest.console.exe, TestPluginDiscoverer: Failed to get types from assembly 'NUnit3.TestAdapter, Version=3.15.1.0, Culture=neutral, PublicKeyToken=4cb40d35494691ac'.  Skipping test extension scan for this assembly.  Error: System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.
   at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
   at System.Reflection.RuntimeModule.GetTypes()
   at System.Reflection.Assembly.GetTypes()
   at Microsoft.VisualStudio.TestPlatform.Common.ExtensionFramework.TestPluginDiscoverer.GetTestExtensionsFromAssembly[TPluginInfo,TExtension](Assembly assembly, Dictionary`2 pluginInfos)
TpTrace Warning: 0 : 22240, 4, 2019/11/06, 17:16:05.318, 3276125107286, vstest.console.exe, LoaderExceptions: System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
File name: 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
   at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.Assembly.Load(AssemblyName assemblyRef)
   at Microsoft.VisualStudio.TestPlatform.Common.ExtensionFramework.TestPluginCache.CurrentDomainAssemblyResolve(Object sender, AssemblyResolveEventArgs args)
   at Microsoft.VisualStudio.TestPlatform.PlatformAbstractions.PlatformAssemblyResolver.AssemblyResolverEvent(Object sender, Object eventArgs)
   at System.AppDomain.OnAssemblyResolveEvent(RuntimeAssembly assembly, String assemblyFullName)

[singhsarab]

@jnm2 Are the adapters present in the directory you are running the cli runner from ?

/testadapterpath should be used only when you are want some specific adapter location to be used for the adapters. In geenral, most .NET adapters get copied next to the test dll and you need not specify it.

What happens when you use /inisolation flag ?

[jnm2]

Are the adapters present in the directory you are running the cli runner from ?

Yes, I checked this. What's more, you can see VSTest finding and loading the adapter DLL and failing while doing so in the first line of the log quote above:

vstest.console.exe, TestPluginDiscoverer: Failed to get types from assembly 'NUnit3.TestAdapter, Version=3.15.1.0, Culture=neutral, PublicKeyToken=4cb40d35494691ac'. Skipping test extension scan for this assembly. Error: System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.

/testadapterpath should be used only when you are want some specific adapter location to be used for the adapters.

This parameter is hardcoded by the Azure Pipelines VSTest task.

What happens when you use /inisolation flag ?

I'm not at my machine right now. @JohanLarsson can you try this?

Regardless of the above, it seems like there is a deeper problem in VSTest if changing either the /logger parameter or the /testadapterpath parameter causes System.Runtime to fail to be found.

[JohanLarsson]

I tried with:

pool:
  vmImage: 'windows-latest'

variables:
  buildConfiguration: 'Release'

steps:

- task: VSBuild@1
  displayName: Build
  inputs:
    configuration: '$(buildConfiguration)'
    msbuildArgs: '/restore'

- task: VSTest@2
  displayName: Test
  inputs:
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'
    runTestsInIsolation: true

But it was hooray for wrong reasons, no tests run. https://dev.azure.com/DotNetAnalyzers/PropertyChangedAnalyzers/_build/results?buildId=581

[jnm2]

Cool, runTestsInIsolation: true was translated to /InIsolation: https://dev.azure.com/DotNetAnalyzers/PropertyChangedAnalyzers/_build/results?buildId=581&view=logs&jobId=12f1170f-54f2-53f3-20dd-22fc7dff55f9&taskId=7bfb7e0f-549d-5929-c603-6e8046613b9e&lineStart=48&lineEnd=48&colStart=1&colEnd=13

@singhsarab The fact is that the adapter DLL beside the test assembly is properly found in all cases. It loads properly when /testadapterpath is specified or when /console:trx is specified, but when both are specified together, it fails to load System.Runtime and therefore fails to load the test adapter.

[singhsarab]

@jnm2 I hope adding /inisolation did help. This makes sure the tests are run in a separate process called testhost, where I assume the runtime error should be getting thrown

Yes, adding /logger:trx possibly is causing the issue as the trxlogger itself is getting loaded in the vstest.console process. If the testadapter or any other assembly is dependent on a different version of the runtime dll they might be causing the failure,

[jnm2]

@singhsarab /inisolation did not help; everything looks the same:

vstest.console.exe "d:\a\1\s\PropertyChangedAnalyzers.Test\bin\Release\net472\PropertyChangedAnalyzers.Test.dll"
/Settings:"d:\a\_temp\qn55nqlalny.tmp.runsettings"
/InIsolation
/Logger:"trx"
/TestAdapterPath:"d:\a\1\s"
Starting test execution, please wait...
No test is available in d:\a\1\s\PropertyChangedAnalyzers.Test\bin\Release\net472\PropertyChangedAnalyzers.Test.dll.

I can check to see if anything is changed in the diagnostic log, but I'm going to guess that it's still failing to load System.Runtime.

Yes, adding /logger:trx possibly is causing the issue as the trxlogger itself is getting loaded in the vstest.console process. If the testadapter or any other assembly is dependent on a different version of the runtime dll they might be causing the failure,

Then why does it also work with /logger:trx and without /testadapterpath? The explanation has to explain each scenario:

• /testadapterpath: only: finds the NUnit test adapter and loads it properly
• /logger:trx only: finds the NUnit test adapter and loads it properly
• /logger:trx and /testadapterpath:: finds the NUnit test adapter and fails to load it due to a failure to load System.Runtime

[JohanLarsson]

There is something weird going on. I have several near identical projects with the same yml and csproj. Here are two projects that runs tests: https://dev.azure.com/DotNetAnalyzers/ReflectionAnalyzers https://dev.azure.com/DotNetAnalyzers/WpfAnalyzers

[singhsarab]

@jnm2 Sorry that we lost track of this issue, is it still reproducing for you ?

[jnm2]

I lost track too. @JohanLarsson have you seen this recently?

[JohanLarsson]

Still a repro: https://dev.azure.com/DotNetAnalyzers/PropertyChangedAnalyzers/_build/results?buildId=2476&view=ms.vss-test-web.build-test-results-tab

[JohanLarsson]

A good default would be to fail build if no tests were run. Setting up a VsTest task and expecting no tests to run should be rare.

[nohwnd]

There is an option to fail the build on no tests https://github.com/microsoft/vstest/issues/2707

[AvremelM]

I'm still having this issue.

[jimmycartrette]

Same, cannot stop /Logger:trx and /TestAdapterPath from being added to VSTest@2, which stops tests from working. Reproduces locally, remove either and it works locally.

[AndirNotes]

I've also got into this issue with migrating to 17.5.0 Microsoft.TestPlatform in our Azure DevOps builds and VSTest@2 task. Any workarounds available? @singhsarab? @Evangelink Could you please explain why it's by design? What that mean?

[Nico-1987]

Same issue here, using VSTest@2 i cannot get my tests to execute. We do not pass a TestAdapterPath into VSTest task, VSTest sets this to the build.sourcesdirectory on its own.

Agent executes this: C:\Agents\AG2\_work\_tool\VsTest\17.6.3\x64\tools\net462\Common7\IDE\Extensions\TestPlatform\vstest.console.exe "C:\Agents\AG2\_work\178\s\bin\Release\Addin\Plugin.Tests.dll" /Settings:"C:\Agents\AG2\_work\178\s\Src\Tests.runsettings" /EnableCodeCoverage /Logger:"trx" /TestAdapterPath:"C:\Agents\AG2\_work\178\s" /Platform:x64 That fails, also when running it manually on the build server

Changing the command line to this does execute the tests (removed the /TestAdapterPath C:\Agents\AG2\_work\_tool\VsTest\17.6.3\x64\tools\net462\Common7\IDE\Extensions\TestPlatform\vstest.console.exe "C:\Agents\AG2\_work\178\s\bin\Release\Addin\Plugin.Tests.dll" /Settings:"C:\Agents\AG2\_work\178\s\Src\Tests.runsettings" /EnableCodeCoverage /Logger:"trx" /Platform:x64

Console output for failed run

Microsoft (R) Test Execution Command Line Tool Version 17.6.3 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
No test is available in C:\Agents\AG2\_work\178\s\bin\Release\Addin\Plugin.Tests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

[nohwnd]

@Nico-1987 this is a long running issue in azure action. The bug for it is here https://github.com/microsoft/azure-pipelines-tasks/issues/16915

A workaround is to point the adapter path to an existing folder (e.g. your docs folder) that has no adapters.

[Nico-1987]

Thank you @nohwnd, i actually found that out an hour ago as well, thank you for posting that here! So to confirm, my issue is resolved by passing another location into the pathtoCustomTestAdapters. Indeed the path has to exists.

[lennartb-]

@nohwnd 's suggestion worked in our case. We used $(Agent.TempDirectory) as the empty, existing folder