MSTest.TestAdapter does not get nuget-restored if the target framework is .NETStandard2.0

[pvlakshm]

Description

MSTest.TestAdapter does not get nuget-restored if the target framework is .NETStandard2.0

Steps to reproduce

1. Install Visual Studio 15.5.0 Preview 2
2. Create a new project using File | New Project | Visual C# | .NET Standard | Class Library (.NET Standard)
3. Add a NuGet Reference to the project to MSTest.TestFramework (v1.2.0)
4. Add a NuGet Reference to the project to MSTest.TestAdapter (v1.2.0)
5. Build the project

Expected behavior

No warning. The MSTest packages should get NuGet-restored as expected.

Actual behavior

warning NU1701: Package 'MSTest.TestAdapter 1.2.0' was restored using '.NETFramework,Version=v4.6.1' instead of the project target framework '.NETStandard,Version=v2.0'. This package may not be fully compatible with your project.

AB#1575875

[codito]

It is not possible to author a test project targeting NETStandard. It must target a app runtime (net46, netcoreappX.Y etc.).

[pvlakshm]

Why does the warning show up only for the TestAdapter package (and not for the TestFramework package)?

[codito]

MSTest.TestFramework lists following as supported targets:

• net45
• netstandard1.0
• uap10.0

The netstandard1.0 target is used for netcoreapp1.0 test projects and it can be used for netstandard1.x projects too.

The test adapter on the other hand lists support for following:

• net45
• netcoreapp1.0
• uap10.0

[pvlakshm]

OK, closing this issue.

[dotMorten]

@pvlakshm Could we reopen this issue? I have a use-case for using the TestAdapter as .NET Standard, when executing tests on Xamarin. I've created my own port of MSTest.TestAdapter (https://www.nuget.org/packages/MSTestX.TestAdapter/) for this purpose, but would prefer just referencing the official package rather than forking and having to attempt to keep in-sync (it compiles as is, so very little effort on your part).

[dotMorten]

It is not possible to author a test project targeting NETStandard. It must target a app runtime (net46, netcoreappX.Y etc.).

This makes no sense to me. That restriction applies to executables, but not class libraries. Sure in the end you'd want to execute your unit tests on a specific runtime, but there's no reason why we can't have unit tests (and the adapter) be .NET Standard.

[jeromelaban]

I'm blocked by this one as well in the context of WebAssembly tests. netstandard2.0 is the only possible target at the time for this kind of environment.

[nimisha84]

I am blocked too. MSTest.testAdapter is not supported for .Net Standard 2.0 and I have a class lib project which needs to run these tests.

[ionsphere]

I'm blocked. Unit tests are supposed to test my ClassLibraries. All the libraries are targeting .NETStandard 2.0. Then why should a Unit test target anything else?

Voting for REOPEN

[dotMorten]

@ionsphere You need a runtime to run your tests on. Do you want to run your tests on .NET Framework, .NET Core, Mono, UWP .NET Core, or... and which version of any of those? There's not such thing as a .NET Standard app, and thus there are no such thing as a .NET Standard Unit Test. You have to the very least reference your class library from a runtime-specific project and execute it in that context.

[ionsphere]

If I have a library that I want to make sure runs on any NET Standard, why shouldn’t it be possible? Isn’t the point of having a reference in VisualStudio stating “NETStandardLibrary 2.0” means all applicable references of that library set? And if so, how testing on NET Core 2.2 proves that the library can run on any NET Standard? There is quite a chance that it may end up requiring NET Framework 4.7.2 with this kind flaky testing.

[ionsphere]

A unit test is not only checking the internal stability, but also the surface of exposed components. How can I be sure that someone’s library while using mine can be NET Standard? How can I verify that all the interfaces and objects in my requirements are implemented under all NET Standard implementations. (Notice, there are interfaces that have no implementing classes under NET Standard and a unit test should hint and help avoiding it) Are you suggesting that with NET Standard I’m forced to have multitude of Unit Test projects each referencing different runtimes, while NET Standard should be a guaranteed testable subset of those runtimes? You know, that NET Core > NET Standard, that NET Framework > NET Standard and that (NET Core ∩ NET Framework) != NET Standard.

[ionsphere]

And I get the point of confusion. Unit Test project should target NETStandard because I want the code to compile under strict limitations of NETStandard. But Unit Test project should run in some runtime compatible with that Standard. As VisualStudio doesn’t seem to have this distinction, then it should be possible to have NETStandard Unit Test and let vstest or dotnet-test pick any installed one with minimal compatibility.

[jayaranigarg]

As part of CPS change, we are trying to consolidate the adapter and framework nuget packages. We will try if this too can be fixed as part of that.

[penfold]

Any progress?

I have .net core 3 web app which has all of its libraries in .net standard 2.1. I really need to be able to test at .net standard 2.1.

Is there a workaround for this?

[abatishchev]

Same here, two years later. Any change to get it fixed? Any known workarounds?

I have a multi-target library and 2 sets of tests: .NET 4.6.2 and .NET Core 2.0.

[MrHinsh]

I to have the same problem. I have a .NET Standard Assembly that I want to write Unit Tests against. Why cant i?

[abatishchev]

The fact this know bug is open for 2+ years and kept without movement is quite frustrating.

[mygithub07]

I am having similar issue where MSTest.TestAdapter loses reference after I create an executable from C# project and so when I run dot exe I get error . The VM where I am running this does not have VS and code base. Requirement is that Exe should include everything .

When I run on local , tests run fine , directly through VS (.NET Framework 4.7.2) or through all included dot exe

No test is available in C:\Users\..\Desktop\MyExeName.exe. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

I am running exe through VSTest.console on this VM

How can I make sure that MSTest tests are discovered through dot exe ?

[redwyre]

I got stuck for a while on this, if .net standard is not supported for test libraries then can we get a better error message that states as such?

[dotMorten]

@redwyre for what it's worth I just updated my test adapter with the latest 2.1.0 code base, if you need it: https://www.nuget.org/packages/MSTestX.TestAdapter/ While I mainly make it to be used with the unit test runner, it should work for anything (just note that it can't do deployment)

[dotMorten]

@MrHinsh

I have a .NET Standard Assembly that I want to write Unit Tests against. Why cant i?

Of course you can. But your unit test project MUST target a framework. Nothing can run as ".NET Standard". That's merely an API contract, not an actual runtime. You'll have to decide to execute the tests on top of .NET Framework, .NET Core or Mono etc. (or all of the above - MSTest does support multi targeting just fine!)

My port to .NET Standard was mainly done so I could run the executor and communications classes on Android and iOS (and thus vstest.console is actually able to talk to my test runner app as the same communication classes are used). But I still have to run the tests themselves in the context of a runtime, in this case the Mono-Xamarin runtime.

[mygithub07]

As for my comment above , I was able to ran my test. The issue actually was that I needed to provide MSTest.TestAdapter package dll through below mentioned config in .runsettings file . I was running test using vstest.console and I used /Settings:[file name] option on command line Here is the config I used

<MSTest>
.....
<AssemblyResolution>
      <Directory path="D:\myfolder\bin\" includeSubDirectories="false"/>
    </AssemblyResolution>
...
<MSTest>

[abatishchev]

But your unit test project MUST target a framework. Nothing can run as ".NET Standard".

Hmm, no. It's not how it works. Test project produces merely a DLL. VSTest.Console.exe is the EXE that runs that DLL. Isn't it?

[mygithub07]

yes. I was targeting framework 4.7.2 and was using vstest.console to run my exe and and provided above mentioned config.

[dotMorten]

VSTest.Console.exe is the EXE that runs that DLL. Isn't it?

Well yes and no. It might start a separate process and run it (a good example is UWP which runs completely isolated in a window). The console then just connects via a socket to control and monitor the test run. One of the things an adapter can do is actually deploy and launch the remote process. An more obvious example is when we used to be able to do it on Windows Phone where it would literally remote deploy and connect to the device. Obviously vstest.console isn't the process executing the tests

[nohwnd]

@dotMorten I think I am still missing the point here. If my test project is netstandard, who will execute the tests? 🙂 I actually tried to move the mstest to netstandard earlier this month, but to do it properly I would need to move the ObjectModel to netstandard1.0 as well (or stick with the very old version it is using now).

Are the changes in your repo preventing some of our other scenarios from running? If not please PR so we can possibly get it merged.

(I am on twitter as @nohwnd and I would love to discuss this with you a bit further, my DMs are open. I have whole two-pager rant about the reasoning around why the projects are currently netcoreapp, and when it might be beneficial to have the project as netstandard, but I don't want to pollute the issue here with it, and because my partial understanding would be a source of confusion here).

[dotMorten]

@nohwnd perhaps I should put it a different way: I really don't need .net standard support. What I need is xamarin.android and xamarin.ios and Tizen support. But then I might as well just use .net standard library. I'll then be referencing mstest in my Xamarin app and use it to do all the test execution, aborts, timeouts, socket communication etc. So yes there's sort of a 3rd non-standard library that acts as the actual app that picks a runtime.

[abatishchev]

Any updated on this, please? I drags on, and on, and on.

Warning NU1701: Package 'MSTest.TestAdapter 2.1.2' was restored using '.NETFramework,Version=v4.6.1, .NETFramework,Version=v4.6.2, .NETFramework,Version=v4.7, .NETFramework,Version=v4.7.1, .NETFramework,Version=v4.7.2, .NETFramework,Version=v4.8' instead of the project target framework '.NETStandard,Version=v2.1'. This package may not be fully compatible with your project.

[Haplois]

We added NETStandard2.0 support into the Test Platform. The next step is to add it to TestFX in upcoming previews of Visual Studio 16.9.