vstest icon indicating copy to clipboard operation
vstest copied to clipboard

Published 20 hours ago • verified publisher icon microsoft

VS2017 build tools: UWP MsTest, Error: No suitable test runtime provider found for this run.

Open ssrakesh opened this issue 6 years ago • 41 comments

Description

Not able to run mstest (v2?) in command line for UWP, getting error "No suitable test runtime provider found for this run."

Steps to reproduce

Created a project by selecting "UnitTest App (Universal Windows)", selected configuration Release and x86 copied the project to Build machine, Build the project (successfully) using command (after restoring Nuget packages) "C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\15.0\Bin\msbuild.exe" /t:Rebuild /p:Configuration="Release" /p:Platform="x86" /verbosity:normal UnitTestProject2.sln"

Run the vstest console command (-Does not work-Error: No suitable test runtime provider found for this run) "C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" "D:\Jenkins\workspace\2018 - UnitTest\UnitTestApp1\AppPackages\UnitTestProject2_1.0.0.0_x86_Test\UnitTestProject2_1.0.0.0_x86.appx" /Framework:FrameworkUap10 /TestAdapterPath:"D:\Jenkins\workspace\2018 - UnitTest\UnitTestApp1\bin\x86\Release" /diag:d:\diag1.txt

Run the vstest console command (-Does not work-Error: No suitable test runtime provider found for this run) "C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" "D:\Jenkins\workspace\2018 - UnitTest\UnitTestApp1\bin\x86\Release\UnitTestProject2.build.appxrecipe" /Framework:FrameworkUap10 /TestAdapterPath:"D:\Jenkins\workspace\2018 - UnitTest\UnitTestApp1\bin\x86\Release"

Expected behavior

Run unit testcases and possibly get the coverage

Actual behavior

Error: No suitable test runtime provider found for this run

Environment

Project created using VS2017 Pro v15.7.4 Build machine with VS2017 build tools v15.7.4, Agent for VS2017 v15.7.4 .cer file imported to local machine->Trusted root certification authority

Diagnostic logs

TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.684, 512874110761, vstest.console.exe, Using .Net Framework version:Uap,Version=v10.0 TpTrace Verbose: 0 : 9100, 1, 2018/06/20, 17:34:34.699, 512874150632, vstest.console.exe, TestPluginCache: Discovering the extensions using extension path. TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.699, 512874163404, vstest.console.exe, AssemblyResolver: Microsoft.TestPlatform.Extensions.BlameDataCollector: Resolving assembly. TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.715, 512874187130, vstest.console.exe, AssemblyResolver: Microsoft.TestPlatform.Extensions.BlameDataCollector: Resolved assembly. TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.715, 512874194714, vstest.console.exe, AssemblyResolver: Microsoft.TestPlatform.Extensions.EventLogCollector: Resolving assembly. TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.715, 512874201358, vstest.console.exe, AssemblyResolver: Microsoft.TestPlatform.Extensions.EventLogCollector: Resolved assembly. TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.715, 512874203630, vstest.console.exe, AssemblyResolver: Microsoft.TestPlatform.TestHostRuntimeProvider: Resolving assembly. TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.715, 512874208720, vstest.console.exe, AssemblyResolver: Microsoft.TestPlatform.TestHostRuntimeProvider: Resolved assembly. TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.715, 512874214648, vstest.console.exe, AssemblyResolver: Microsoft.VisualStudio.TestPlatform.Extensions.CodedWebTestAdapter: Resolving assembly. TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.715, 512874219435, vstest.console.exe, AssemblyResolver: Microsoft.VisualStudio.TestPlatform.Extensions.CodedWebTestAdapter: Resolved assembly. TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.730, 512874231594, vstest.console.exe, AssemblyResolver: Microsoft.VisualStudio.TestPlatform.Extensions.GenericTestAdapter: Resolving assembly. TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.730, 512874236518, vstest.console.exe, AssemblyResolver: Microsoft.VisualStudio.TestPlatform.Extensions.GenericTestAdapter: Resolved assembly. TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.730, 512874238295, vstest.console.exe, AssemblyResolver: Microsoft.VisualStudio.TestPlatform.Extensions.OrderedTestAdapter: Resolving assembly. TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.730, 512874243289, vstest.console.exe, AssemblyResolver: Microsoft.VisualStudio.TestPlatform.Extensions.OrderedTestAdapter: Resolved assembly. TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.730, 512874245033, vstest.console.exe, AssemblyResolver: Microsoft.VisualStudio.TestPlatform.Extensions.TmiAdapter: Resolving assembly. TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.730, 512874247512, vstest.console.exe, AssemblyResolver: Microsoft.VisualStudio.TestPlatform.Extensions.TmiAdapter: Resolved assembly. TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.777, 512874412487, vstest.console.exe, AssemblyResolver: Microsoft.VisualStudio.TestPlatform.Extensions.Trx.TestLogger: Resolving assembly. TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.793, 512874418328, vstest.console.exe, AssemblyResolver: Microsoft.VisualStudio.TestPlatform.Extensions.Trx.TestLogger: Resolved assembly. TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.793, 512874425250, vstest.console.exe, AssemblyResolver: Microsoft.VisualStudio.TestPlatform.Extensions.VSTestIntegration: Resolving assembly. TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.793, 512874430445, vstest.console.exe, AssemblyResolver: Microsoft.VisualStudio.TestPlatform.Extensions.VSTestIntegration: Resolved assembly. TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.793, 512874453124, vstest.console.exe, AssemblyResolver: Microsoft.VisualStudio.TestPlatform.Extensions.WebTestAdapter: Resolving assembly. TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.793, 512874457999, vstest.console.exe, AssemblyResolver: Microsoft.VisualStudio.TestPlatform.Extensions.WebTestAdapter: Resolved assembly. TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.793, 512874459898, vstest.console.exe, AssemblyResolver: Microsoft.VisualStudio.TestTools.CppUnitTestFramework.ComInterfaces: Resolving assembly. TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.793, 512874464699, vstest.console.exe, AssemblyResolver: Microsoft.VisualStudio.TestTools.CppUnitTestFramework.ComInterfaces: Resolved assembly. TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.809, 512874466905, vstest.console.exe, AssemblyResolver: Microsoft.VisualStudio.TestTools.CppUnitTestFramework.CppUnitTestExtension: Resolving assembly. TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.809, 512874471982, vstest.console.exe, AssemblyResolver: Microsoft.VisualStudio.TestTools.CppUnitTestFramework.CppUnitTestExtension: Resolved assembly. TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.809, 512874475400, vstest.console.exe, AssemblyResolver: Microsoft.VisualStudio.TestTools.DataCollection.MediaRecorder.Model: Resolving assembly. TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.812, 512874480498, vstest.console.exe, AssemblyResolver: Microsoft.VisualStudio.TestTools.DataCollection.MediaRecorder.Model: Resolved assembly. TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.812, 512874481935, vstest.console.exe, AssemblyResolver: Microsoft.VisualStudio.TestTools.DataCollection.VideoRecorderCollector: Resolving assembly. TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.812, 512874487113, vstest.console.exe, AssemblyResolver: Microsoft.VisualStudio.TestTools.DataCollection.VideoRecorderCollector: Resolved assembly. TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.812, 512874490977, vstest.console.exe, AssemblyResolver: Microsoft.VisualStudio.TraceDataCollector: Resolving assembly. TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.812, 512874496319, vstest.console.exe, AssemblyResolver: Microsoft.VisualStudio.TraceDataCollector: Resolved assembly. TpTrace Verbose: 0 : 9100, 1, 2018/06/20, 17:34:34.828, 512874532174, vstest.console.exe, TestPluginCache: Discovered the extensions using extension path ''. TpTrace Verbose: 0 : 9100, 1, 2018/06/20, 17:34:34.828, 512874534906, vstest.console.exe, TestPluginCache: Discoverers are ''. TpTrace Verbose: 0 : 9100, 1, 2018/06/20, 17:34:34.828, 512874535110, vstest.console.exe, TestPluginCache: Executors are ''. TpTrace Verbose: 0 : 9100, 1, 2018/06/20, 17:34:34.828, 512874535251, vstest.console.exe, TestPluginCache: Setting providers are ''. TpTrace Verbose: 0 : 9100, 1, 2018/06/20, 17:34:34.828, 512874535380, vstest.console.exe, TestPluginCache: Loggers are ''. TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.828, 512874567051, vstest.console.exe, RunTestsArgumentProcessor:Execute: Test run is starting. TpTrace Verbose: 0 : 9100, 1, 2018/06/20, 17:34:34.828, 512874567289, vstest.console.exe, RunTestsArgumentProcessor:Execute: Queuing Test run. TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.828, 512874572573, vstest.console.exe, TestRequestManager.RunTests: run tests started. TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.843, 512874585273, vstest.console.exe, Determined framework for all sources: Uap,Version=v10.0 TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.843, 512874591204, vstest.console.exe, Determined platform for all sources: X86 TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.843, 512874600213, vstest.console.exe, Compatible sources list : TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.843, 512874600412, vstest.console.exe, D:\Jenkins\workspace\2018 -UnitTest\UnitTestApp1\bin\x86\Release\UnitTestProject2.build.appxrecipe TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.859, 512874626796, vstest.console.exe, InferRunSettingsHelper.IsTestSettingsEnabled: Unable to navigate to RunSettings/MSTest. Current node: RunSettings TpTrace Verbose: 0 : 9100, 1, 2018/06/20, 17:34:34.859, 512874658286, vstest.console.exe, TestPluginCache: Update extensions started. Skip filter = False TpTrace Verbose: 0 : 9100, 1, 2018/06/20, 17:34:34.859, 512874660305, vstest.console.exe, TestPluginCache: Using directories for assembly resolution 'D:\Jenkins\workspace\2018 -UnitTest\UnitTestApp1\bin\x86\Release'. TpTrace Verbose: 0 : 9100, 1, 2018/06/20, 17:34:34.859, 512874660510, vstest.console.exe, TestPluginCache: Updated the available extensions to 'D:\Jenkins\workspace\2018 -UnitTest\UnitTestApp1\bin\x86\Release\Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll'. TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.890, 512874721970, vstest.console.exe, AssemblyResolver: Microsoft.TestPlatform.TestHostRuntimeProvider, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a: Resolving assembly. TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.890, 512874723770, vstest.console.exe, AssemblyResolver: Microsoft.TestPlatform.TestHostRuntimeProvider, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a: Resolved assembly. TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.890, 512874725811, vstest.console.exe, TestPluginManager.CreateTestExtension: Attempting to load test extension: Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Hosting.DefaultTestHostManager TpTrace Information: 0 : 9100, 1, 2018/06/20, 17:34:34.890, 512874728130, vstest.console.exe, TestPluginManager.CreateTestExtension: Attempting to load test extension: Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Hosting.DotnetTestHostManager TpTrace Error: 0 : 9100, 1, 2018/06/20, 17:34:34.890, 512874730164, vstest.console.exe, TestPlatform.CreateTestRunRequest: No suitable testHostProvider found for runsettings : <RunSettings> <RunConfiguration> <ResultsDirectory>D:\Jenkins\workspace\2018 -UnitTest\UnitTestApp1\TestResults</ResultsDirectory> <TargetPlatform>X86</TargetPlatform> <TargetFrameworkVersion>Uap,Version=v10.0</TargetFrameworkVersion> <TestAdaptersPaths>D:\Jenkins\workspace\2018 -UnitTest\UnitTestApp1\bin\x86\Release</TestAdaptersPaths> <DesignMode>False</DesignMode> <CollectSourceInformation>False</CollectSourceInformation> </RunConfiguration> <LoggerRunSettings> <Loggers> <Logger friendlyName="Console" uri="logger://microsoft/TestPlatform/ConsoleLogger/v1" assemblyQualifiedName="Microsoft.VisualStudio.TestPlatform.CommandLine.Internal.ConsoleLogger, vstest.console, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" codeBase="C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" enabled="True" /> </Loggers> </LoggerRunSettings> </RunSettings> TpTrace Error: 0 : 9100, 1, 2018/06/20, 17:34:34.890, 512874736765, vstest.console.exe, TestRequestManager.RunTests: failed to run tests: Microsoft.VisualStudio.TestPlatform.ObjectModel.TestPlatformException: No suitable test runtime provider found for this run. at Microsoft.VisualStudio.TestPlatform.Client.TestPlatform.ThrowExceptionIfTestHostManagerIsNull(ITestRuntimeProvider testHostManager, String settingXml) at Microsoft.VisualStudio.TestPlatform.Client.TestPlatform.CreateTestRunRequest(IRequestData requestData, TestRunCriteria testRunCriteria, TestPlatformOptions options) at Microsoft.VisualStudio.TestPlatform.CommandLine.TestPlatformHelpers.TestRequestManager.RunTests(IRequestData requestData, TestRunCriteria testRunCriteria, ITestRunEventsRegistrar testRunEventsRegistrar, TestPlatformOptions options) TpTrace Error: 0 : 9100, 1, 2018/06/20, 17:34:34.890, 512874737881, vstest.console.exe, ExecuteArgumentProcessor: failed to execute argument process: Microsoft.VisualStudio.TestPlatform.ObjectModel.TestPlatformException: No suitable test runtime provider found for this run. at Microsoft.VisualStudio.TestPlatform.Client.TestPlatform.ThrowExceptionIfTestHostManagerIsNull(ITestRuntimeProvider testHostManager, String settingXml) at Microsoft.VisualStudio.TestPlatform.Client.TestPlatform.CreateTestRunRequest(IRequestData requestData, TestRunCriteria testRunCriteria, TestPlatformOptions options) at Microsoft.VisualStudio.TestPlatform.CommandLine.TestPlatformHelpers.TestRequestManager.RunTests(IRequestData requestData, TestRunCriteria testRunCriteria, ITestRunEventsRegistrar testRunEventsRegistrar, TestPlatformOptions options) at Microsoft.VisualStudio.TestPlatform.CommandLine.TestPlatformHelpers.TestRequestManager.RunTests(TestRunRequestPayload testRunRequestPayload, ITestHostLauncher testHostLauncher, ITestRunEventsRegistrar testRunEventsRegistrar, ProtocolConfig protocolConfig) at Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.RunTestsArgumentExecutor.RunTests(IEnumerable`1 sources) at Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.RunTestsArgumentExecutor.Execute() at Microsoft.VisualStudio.TestPlatform.CommandLine.Executor.ExecuteArgumentProcessor(IArgumentProcessor processor, Int32& exitCode) TpTrace Verbose: 0 : 9100, 1, 2018/06/20, 17:34:34.890, 512874740610, vstest.console.exe, Executor.Execute: Exiting with exit code of 1

ssrakesh avatar Jun 20 '18 12:06 ssrakesh

Also Installed Agents for Visual Studio 2017, and executed vstest.console.exe from the same. still same error occurs

ssrakesh avatar Jun 21 '18 10:06 ssrakesh

By comparing diag log from the vstest in VS2017 Pro system and build tools system, I copied Microsoft.VisualStudio.UwpTestHostRuntimeProvider.Deployment, Microsoft.VisualStudio.UwpTestHostRuntimeProvider, Microsoft.VisualStudio.Telemetry to Build tools 2017 system. Then I got the error "This operation requires installation of Visual Studio with the Universal Windows Platform development workload selected."

ssrakesh avatar Jun 22 '18 04:06 ssrakesh

@ssrakesh

By looking at log from the VS2017 Pro system, I copied Microsoft.VisualStudio.UwpTestHostRuntimeProvider.Deployment, Microsoft.VisualStudio.UwpTestHostRuntimeProvider, Microsoft.VisualStudio.Telemetry to Build tools 2017 system. Then I got the error "This operation requires installation of Visual Studio with the Universal Windows Platform development workload selected."

Please don't do this which can corrupt your Buildtools installation.

No suitable test runtime provider found for this run

This error occurs when UWP workload wasn't installed. or UWP workload didn't ship the runtime provider which is required for vstest.console to run UWP tests.

Can you share what are the workloads have you installed in buildtools?

smadala avatar Jun 23 '18 12:06 smadala

I have already installed UWP work load in build tools, and UwpTestHostRuntimeProvider was missing from it, not sure if the same dll is responsible to resolve .appxreceipe in build tools. I also tried by reinstalling UWP workload by command prompt. workloads installation

ssrakesh avatar Jun 25 '18 04:06 ssrakesh

@ssrakesh UWP workload should drop the runtime provider in TestPlatform folder. We are working with UWP team to find why the runtime provider assembly not available in build tools. I will let you know as soon as I have more info .

smadala avatar Jun 27 '18 13:06 smadala

Thanks a lot

ssrakesh avatar Jun 28 '18 06:06 ssrakesh

btw, does this vstest project standalone has UWP testplatform to test .appxreciepe? or this requires VS?

ssrakesh avatar Jul 01 '18 16:07 ssrakesh

@ssrakesh

btw, does this vstest project standalone has UWP testplatform to test .appxreciepe? or this requires VS?

No, vstest.console required VS installation to run UWP tests.

smadala avatar Jul 02 '18 14:07 smadala

As @smadala mentioned some of the components required to light up this scenario are not part of the Build Tools SKU. This will currently not work (as you have observed) and getting those components into Build Tools is not in our immediate roadmap. Please let us know if this is a blocker for you?

pvlakshm avatar Jul 19 '18 10:07 pvlakshm

Hi, Yes currently this issue is a blocking one, bz we don't want to install full VS 2017 in build PC. So we cannot run our unit test projects and also coverage automatically. so we have to end up doing these manually in developer PC. Also I am not sure even with installing VS2017 Pro, I could run the UWP test cases automatically without login to desktop as UWP test cases runs with UI So developers are forced to go with Team server. Regarding coverage, we are not able to get this working with UWP. in VS2017 pro version. I was in impression that UWP is more preferred development than WPF,

Also this point should either a bug or missing feature, but still the status is marked as "Question"?

ssrakesh avatar Jul 20 '18 06:07 ssrakesh

I am afraid this scenario will not work. Getting this right components packaged into the Build Tools SKU will likely figure for inclusion only in the next planning cycle. I have cleared the "Question" tag.

pvlakshm avatar Jul 20 '18 10:07 pvlakshm

This has action item on UWP team, tracking bug for same internally here

smadala avatar Aug 01 '18 23:08 smadala

Which version of Visual Studio will work with Microsoft.CodeCoverage 15.8.0? I believe this is the same issue we face when trying to upgrade Microsoft.CodeCoverage from 1.0.3 to 15.8.0 as the Build environment on Azure DevOps for a Hosted VS2017 Agent has VisualStudio/15.8.1+28010.2003 installed.

KarlZ avatar Sep 19 '18 20:09 KarlZ

Is there any update on this issue?

ssrakesh avatar Sep 25 '18 05:09 ssrakesh

@ssrakesh: This is a bigger change and unfortunately this will not be a part of VS 2017. This will be done as part of next VS release.

jayaranigarg avatar Sep 26 '18 08:09 jayaranigarg

At-least Is this going to be included in VS2019? recently there was a roadmap announcement for the same.

ssrakesh avatar Nov 30 '18 06:11 ssrakesh

Does this related to https://github.com/Microsoft/react-native-windows/issues/2031 and https://github.com/Microsoft/testfx/issues/314#issuecomment-444374630, it blocked me with the startup of react-native-windows.

liudonghua123 avatar Dec 05 '18 09:12 liudonghua123

@liudonghua123 : You are trying to access UWP c++ files in build tools. Those are currently not available. Tagging @MSLukeWest

jayaranigarg avatar Dec 05 '18 09:12 jayaranigarg

@jayaranigarg, I followed the instructions of https://github.com/Microsoft/react-native-windows/blob/master/docs/GettingStarted.md, then this error occurs, So the https://github.com/Microsoft/react-native-windows project does not support Windows? 😢

liudonghua123 avatar Dec 06 '18 01:12 liudonghua123

I am afraid this scenario will not work. Getting this right components packaged into the Build Tools SKU will likely figure for inclusion only in the next planning cycle. I have cleared the "Question" tag.

@pvlakshm this is also a major issue for us in our CI environment.

phraemer avatar Mar 27 '19 14:03 phraemer

Anyone made progress on this?

NelsonLamprecht avatar Dec 11 '20 23:12 NelsonLamprecht

@NelsonLamprecht Yes, the most recent release of Visual Studio 2019 supports running UWP unit tests with just the Build Tools sku installed.

MSLukeWest avatar Dec 12 '20 00:12 MSLukeWest

@NelsonLamprecht Yes, the most recent release of Visual Studio 2019 supports running UWP unit tests with just the Build Tools sku installed.

Trying to figure out what to install on our Azure Dev Ops local build agent to support running our UWP unit test via Azure Pipelines: Currently generating "##[error]No suitable test runtime provider found for this run" with vstest.console.exe "/Framework:FrameworkUap10"

--update We tried the Azure Pipelines host but still got the same error message

NelsonLamprecht avatar Dec 12 '20 23:12 NelsonLamprecht

If you're installing the Build Tools sku you will need to include the "Universal Windows Platform build tools" workload. The ID is "Microsoft.VisualStudio.Workload.UniversalBuildTools".

MSLukeWest avatar Dec 14 '20 19:12 MSLukeWest

So years later and this is still open? How exactly should I be running UWP tests from the command line?

My system:

  • Docker Windows 10 image (windowsservercore-ltsc2019), with the following workloads:
  • Microsoft.VisualStudio.Workload.MSBuildTools
  • Microsoft.VisualStudio.Workload.ManagedDesktopBuildTools
  • Microsoft.VisualStudio.Workload.UniversalBuildTools
  • Microsoft.VisualStudio.Component.VC.CoreBuildTools

I've tried to get this working with dotnet test to no avail, and then saw a bunch of people saying that we need to use vstest.console to run UWP tests... so currently, I am running this command line:

vstest.console /Logger:trx /Enablecodecoverage /Platform:x64 /Framework:FrameworkUAP10 <my project's appxrecipe>

This command used to give me an error in the diagnostic output that I was missing Microsoft.VisualStudio.TestPlatform, but since adding that as a NuGet package in my test projects I no longer get that error. Now, running the command in diagnostic mode, I am finding a bunch of errors in the output diagnostic results related to missing Microsoft.VisualStudio.TestPlatform.Fakes.

TpTrace Information: 0 : 1520, 1, 2021/12/03, 12:52:45.641, 2091976380, vstest.console, Failed to load assembly Microsoft.VisualStudio.TestPlatform.Fakes, Version=17.0.0.0, Culture=neutral. Reason:System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.VisualStudio.TestPlatform.Fakes, Version=17.0.0.0, Culture=neutral' or one of its dependencies. The system cannot find the file specified. File name: 'Microsoft.VisualStudio.TestPlatform.Fakes, Version=17.0.0.0, Culture=neutral' at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks) at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks) at System.Reflection.Assembly.Load(AssemblyName assemblyRef) at Microsoft.VisualStudio.TestPlatform.Common.Utilities.FakesUtilities.LoadTestPlatformAssembly()

And the final stack trace that I get at the bottom of the diagnostic output is:

TpTrace Error: 0 : 1520, 1, 2021/12/03, 12:52:45.693, 2092528504, vstest.console, TestRequestManager.RunTests: failed to run tests: Microsoft.VisualStudio.TestPlatform.ObjectModel.TestPlatformException: No suitable test runtime provider found for this run. at Microsoft.VisualStudio.TestPlatform.Client.TestPlatform.ThrowExceptionIfTestHostManagerIsNull(ITestRuntimeProvider testHostManager, String settingXml) at Microsoft.VisualStudio.TestPlatform.Client.TestPlatform.CreateTestRunRequest(IRequestData requestData, TestRunCriteria testRunCriteria, TestPlatformOptions options) at Microsoft.VisualStudio.TestPlatform.CommandLine.TestPlatformHelpers.TestRequestManager.RunTests(IRequestData requestData, TestRunCriteria testRunCriteria, ITestRunEventsRegistrar testRunEventsRegistrar, TestPlatformOptions options) TpTrace Error: 0 : 1520, 1, 2021/12/03, 12:52:45.693, 2092534013, vstest.console, ExecuteArgumentProcessor: failed to execute argument process: Microsoft.VisualStudio.TestPlatform.ObjectModel.TestPlatformException: No suitable test runtime provider found for this run. at Microsoft.VisualStudio.TestPlatform.Client.TestPlatform.ThrowExceptionIfTestHostManagerIsNull(ITestRuntimeProvider testHostManager, String settingXml) at Microsoft.VisualStudio.TestPlatform.Client.TestPlatform.CreateTestRunRequest(IRequestData requestData, TestRunCriteria testRunCriteria, TestPlatformOptions options) at Microsoft.VisualStudio.TestPlatform.CommandLine.TestPlatformHelpers.TestRequestManager.RunTests(IRequestData requestData, TestRunCriteria testRunCriteria, ITestRunEventsRegistrar testRunEventsRegistrar, TestPlatformOptions options) at Microsoft.VisualStudio.TestPlatform.CommandLine.TestPlatformHelpers.TestRequestManager.RunTests(TestRunRequestPayload testRunRequestPayload, ITestHostLauncher testHostLauncher, ITestRunEventsRegistrar testRunEventsRegistrar, ProtocolConfig protocolConfig) at Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.RunTestsArgumentExecutor.RunTests(String runSettings) at Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.RunTestsArgumentExecutor.Execute() at Microsoft.VisualStudio.TestPlatform.CommandLine.Executor.ExecuteArgumentProcessor(IArgumentProcessor processor, Int32& exitCode)

Any pointers or help in debugging this would be appreciated- this is a part of an automated build system running in docker containers, which precludes doing things like installing visual studio or any UI-related apps.

aronweiler avatar Dec 03 '21 21:12 aronweiler

Update- I made some progress on this over the last day.

Looking through all of the referenced files in my build system, I have found the Microsoft.VisualStudio.TestPlatform.Fakes.dll, it is in the <nuget packge folder>\microsoft.testplatform\17.0.0\tools\net451\Common7\IDE\Extensions\TestPlatform\ directory, which comes as part of the Microsoft.TestPlatform NuGet package.

I attempted to manually place the that file (Microsoft.VisualStudio.TestPlatform.Fakes.dll) in a directory that was being searched by VSTest.Console:

(majority of the stack trace omitted for brevity)

Microsoft.VisualStudio.TestPlatform.Fakes, Version=17.0.0.0, Culture=neutral: Searching in: '**C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\Common7\IDE\Extensions\TestPlatform\Extensions**'.
...
Microsoft.VisualStudio.TestPlatform.Fakes, Version=17.0.0.0, Culture=neutral: Searching in: '**C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\Common7\IDE\Extensions\TestPlatform**'.
...

You can see above that there are a couple directories being searched- I placed the Microsoft.VisualStudio.TestPlatform.Fakes.dll in both, just to be sure it was picked up.

This is an issue in itself, as I shouldn't have to manually copy a DLL from the NuGet package to a tools folder on my build machine.

Once that was done, there are no more missing library issues- there is now just the following exception:

TpTrace Error: 0 : 1484, 5, 2021/12/05, 22:53:03.728, 17143735446, vstest.console, ProxyExecutionManager.StartTestRun: Failed to start test run: Microsoft.VisualStudio.TestPlatform.ObjectModel.TestPlatformException: Failed to initialize client proxy: could not connect to test process.
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyOperationManager.ThrowExceptionOnConnectionFailure(Int32 connTimeout)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyOperationManager.SetupChannel(IEnumerable`1 sources, String runSettings)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyExecutionManager.StartTestRun(TestRunCriteria testRunCriteria, ITestRunEventsHandler eventHandler)

Still looking for a real command-line test solution for UWP projects...

aronweiler avatar Dec 06 '21 07:12 aronweiler

@aronweiler - The likely cause of this issue is that the Universal build tools workload (ID: "Microsoft.VisualStudio.Workload.UniversalBuildTools") isn't included in your install. You can check this by looking for the assembly <VS Install Dir>\Common7\IDE\Extensions\TestPlatform\Extensions\Microsoft.VisualStudio.UwpTestHostRuntimeProvider.dll. If that assembly isn't on the machine then that means the workload wasn't included.

The message about the missing Microsoft.VisualStudio.TestPlatform.Fakes.dll is just a red herring that can be ignored. That will appear in the log even when running the test works.

MSLukeWest avatar Dec 07 '21 20:12 MSLukeWest

The Microsoft.VisualStudio.Workload.UniversalBuildTools workload is definitely included, and the UwpTestHostRuntimeProvider.dll is present in the following location:

C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\Common7\IDE\Extensions\TestPlatform\Extensions

aronweiler avatar Dec 07 '21 21:12 aronweiler

@aronweiler If the Universal build tools workload is installed you shouldn't be getting that error, please produce a diagnostic log and share it with us so we can take a look. You do this by adding the --diag:<LogFile> parameter to your vstest.console.exe command. (example: "--Diag:c:\foo\bar\log.txt")

MSLukeWest avatar Dec 08 '21 00:12 MSLukeWest

Ok, attached is the diagnostic file from the test command below:

vstest.console /Logger:trx /Platform:x64 /Framework:FrameworkUAP10 OmnitoolUWP\Omnitool.Tests.Unit\bin\x64\Rele ase\Omnitool.Tests.Unit.build.appxrecipe /diag:c:\build\diag.txt

diag.txt

aronweiler avatar Dec 08 '21 23:12 aronweiler