Add initial support for Microsoft.Testing.Platform

[Evangelink]

Fixes #402

[bradwilson]

@Evangelink When building the sample, I get a compile failure that looks like some kind of missing assembly/package reference:

1>------ Build started: Project: xunit-runner-sample, Configuration: Debug Any CPU ------
1>C:\Dev\xunit\visualstudio\samples\xunit-runner-sample\obj\Debug\net462\TestPlatformEntryPoint.cs(12,27,12,34): error CS0234: The type or namespace name 'Testing' does not exist in the namespace 'Microsoft' (are you missing an assembly reference?)
1>C:\Dev\xunit\visualstudio\samples\xunit-runner-sample\obj\Debug\net462\TestPlatformEntryPoint.cs(12,92,12,117): error CS0234: The type or namespace name 'Testing' does not exist in the namespace 'Microsoft' (are you missing an assembly reference?)
1>C:\Dev\xunit\visualstudio\samples\xunit-runner-sample\obj\Debug\net462\TestPlatformEntryPoint.cs(13,9,13,26): error CS0234: The type or namespace name 'Testing' does not exist in the namespace 'Microsoft' (are you missing an assembly reference?)
1>C:\Dev\xunit\visualstudio\samples\xunit-runner-sample\obj\Debug\net462\TestPlatformEntryPoint.cs(14,9,14,26): error CS0234: The type or namespace name 'Testing' does not exist in the namespace 'Microsoft' (are you missing an assembly reference?)
1>C:\Dev\xunit\visualstudio\samples\xunit-runner-sample\obj\Debug\net462\TestPlatformEntryPoint.cs(15,9,15,75): error CS0012: The type 'ITestApplicationBuilder' is defined in an assembly that is not referenced. You must add a reference to assembly 'Microsoft.Testing.Platform, Version=1.0.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
1>C:\Dev\xunit\visualstudio\samples\xunit-runner-sample\obj\Debug\net462\TestPlatformEntryPoint.cs(16,34,16,41): error CS0234: The type or namespace name 'Testing' does not exist in the namespace 'Microsoft' (are you missing an assembly reference?)
1>Done building project "xunit-runner-sample.csproj" -- FAILED.
1>C:\Dev\xunit\visualstudio\samples\xunit-runner-sample\obj\Debug\net6.0\TestPlatformEntryPoint.cs(12,27,12,34): error CS0234: The type or namespace name 'Testing' does not exist in the namespace 'Microsoft' (are you missing an assembly reference?)
1>C:\Dev\xunit\visualstudio\samples\xunit-runner-sample\obj\Debug\net6.0\TestPlatformEntryPoint.cs(12,92,12,117): error CS0234: The type or namespace name 'Testing' does not exist in the namespace 'Microsoft' (are you missing an assembly reference?)
1>C:\Dev\xunit\visualstudio\samples\xunit-runner-sample\obj\Debug\net6.0\TestPlatformEntryPoint.cs(13,9,13,26): error CS0234: The type or namespace name 'Testing' does not exist in the namespace 'Microsoft' (are you missing an assembly reference?)
1>C:\Dev\xunit\visualstudio\samples\xunit-runner-sample\obj\Debug\net6.0\TestPlatformEntryPoint.cs(14,9,14,26): error CS0234: The type or namespace name 'Testing' does not exist in the namespace 'Microsoft' (are you missing an assembly reference?)
1>C:\Dev\xunit\visualstudio\samples\xunit-runner-sample\obj\Debug\net6.0\TestPlatformEntryPoint.cs(15,9,15,75): error CS0012: The type 'ITestApplicationBuilder' is defined in an assembly that is not referenced. You must add a reference to assembly 'Microsoft.Testing.Platform, Version=1.0.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
1>C:\Dev\xunit\visualstudio\samples\xunit-runner-sample\obj\Debug\net6.0\TestPlatformEntryPoint.cs(16,34,16,41): error CS0234: The type or namespace name 'Testing' does not exist in the namespace 'Microsoft' (are you missing an assembly reference?)
1>Done building project "xunit-runner-sample.csproj" -- FAILED.

If I set <EnableXunitRunner>false</EnableXunitRunner>, the compile errors go away. It looks like the hooks for MSBuild are all triggering, but something is missing with references.

[bradwilson]

Looks like it might be package reference related. Converting this:

<PackageReference Include="xunit.runner.visualstudio" Version="2.5.8-pre.2">
  <PrivateAssets>all</PrivateAssets>
  <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
</PackageReference>

to this:

<PackageReference Include="xunit.runner.visualstudio" Version="2.5.8-pre.2" />

fixes the issue. So it looks like setting developmentDependency is probably wrong for .nuspec now. Next question to answer is: Will NuGet pick up changes to this during package upgrade and update the PackageReference appropriately?

[bradwilson]

All three of these variants did not upgrade the package correctly:

1. Package Manager in VS2022
2. dotnet add package xunit.runner.visualstudio --prerelease
3. dotnet outdated -u

That's pretty disappointing. That means we're on a failure path here unless we can somehow convince people to uninstall and then reinstall the package. 😞

[bradwilson]

Also, it looks like you're setting a value in the .targets file (GenerateTestingPlatformEntryPoint) that you depend on in the .props files, which seems like the wrong order. Shouldn't that top PropertyGroup from the .targets file actually be in the .props files?

[bradwilson]

What are the plans for dotnet vstest? It appears that it currently falls back to the old behavior.

[bradwilson]

Why are .NET Framework tests run in x86 by default if the test project is compiled as AnyCPU?

Run tests: 'C:\Dev\xunit\visualstudio\samples\xunit-runner-sample\bin\Debug\net462\xunit-runner-sample.exe' [net462|x86]
Run tests: 'C:\Dev\xunit\visualstudio\samples\xunit-runner-sample\bin\Debug\net6.0\xunit-runner-sample.dll' [net6.0|x64]

[Evangelink]

Hey @bradwilson,

Sorry for the delay in my replies, I am off and had some trouble finding some time to get back here. I'll resume work on 11th so will be fully available for sure at this stage.

So it looks like setting developmentDependency is probably wrong for .nuspec now

I assume that this is telling to msbuild that all deps are only compile time deps and not runtime ones but for the runner we do need the platform + adapter to be available at runtime. I see you have created a ticket on NuGet, I will try to ping some people so we can see how we can unblock this. @baronfel would you have some info or contact about that?

What are the plans for dotnet vstest? It appears that it currently falls back to the old behavior.

As the name implies this is explicitly asking for the vstest mode. This is equivalent to dotnet test without the property TestingPlatformDotnetTestSupport enabled (see https://learn.microsoft.com/dotnet/core/testing/unit-testing-mstest-runner-integrations#dotnet-test---mstest-runner-mode). While you reference the VSTest bridge package this will continue to work but as soon as this deps is dropped this will no longer work.

Our goal is to be able to deprecate VSTest entirely but this will most likely take some time as we will need to have enough people moved and happy on the new platform.

Why are .NET Framework tests run in x86 by default if the test project is compiled as AnyCPU?

We are not doing any change on our side, this seems to be MSBuild defaults that changed between netfx and netcore. @MarcoRossignoli do you know? @baronfel do you know about that? Or do you have someone that might know?

[Evangelink]

Why are .NET Framework tests run in x86 by default if the test project is compiled as AnyCPU?

@MarcoRossignoli just pointed out this ticket https://github.com/dotnet/sdk/issues/3492 that confirms this is the "default" behavior and this is a change linked to the sdk style project.

[bradwilson]

Sorry for the delay in my replies

No worries, this isn't a rush situation. :)

[bradwilson]

As the name implies this is explicitly asking for the vstest mode.

I ask because I use dotnet vstest with the --Parallel switch because it has superior behavior in that mode than just running dotnet test where all the output is interleaved and not combined (and it allows me to be more selective on what gets run). I understand that the new dotnet test output is basically so sparse that the interleaved output problem is gone, though I suspect you will get end user push back about this being too spares for interactive consumption. You should certainly consider my voice at the top of that list. 😄 Losing the ability to be more selective would be unfortunate, though in our case we do still plan to ship our multi-runners in v3 (like the console runner) that would allow us to continue to be more selective.

[baronfel]

@bradwilson .NET 9 preview 2 includes a rework of the test-reporting infrastructure that does show testing progress, as well as ordered test result reporting directly in MSBuild. It's built on a tiny bit of a hack while the MSBuild team works on a proper progress/ephemeral message API for the MSBuild logging infrastructure, after which we should be able to use those new message types to implement nice CLI progress reporting and such. So it's coming, we're working on it :)

[Evangelink]

We definitely want to work on and improve the interactive mode. As Chet is saying we have already started some work and we are also exploring some paths in parallel so we can decide what's best and most evolutive

[bradwilson]

@Evangelink @MarcoRossignoli @baronfel Where is the source for Microsoft.Testing.Extensions.VSTestBridge? I see dependencies on it in https://github.com/microsoft/testfx but I don't see the source.

[bradwilson]

Is the planned IPC for support inside Visual Studio/VS Code also going to be named pipes? If so, what is the equivalent of --internal-msbuild-node <pipename> to support that? If not, what is the planned IPC mechanism and which client class should I be using to support it?

Why are all the types in Microsoft.Testing.Platform.IPC marked as internal?

[bradwilson]

I'm just going to start opening issues with these questions so they're not buried here, and others can contribute (and perhaps get the benefit of the answers).

First set of issues:

https://github.com/microsoft/testfx/issues/2536 ("FailedTestNodeStateProperty should include the ability to report without an exception") https://github.com/microsoft/testfx/issues/2537 ("What is the intended purpose for the hierarchy provided in TestNode?") https://github.com/microsoft/testfx/issues/2538 ("How do I report a test that's not run?") https://github.com/microsoft/testfx/issues/2539 ("Please document the IPC handshake mechanisms and open up the classes under Microsoft.Testing.Platform.IPC")

[bradwilson]

FYI, I have a parallel prototype effort to integrate Microsoft.Testing.Platform directly into v3's existing architecture without using the VSTest adapter: https://github.com/xunit/xunit/tree/microsoft-testing-platform