microsoft
Incompletely configured MTP in `dotnet test` VSTest *silently* doesn't run tests
Describe the bug
If one omits the single critical step of adding e.g. <EnableNUnitRunner>true</EnableNUnitRunner> (for NUnit) in one's test project file, while running in vstest mode of dotnet test due to the absence of the .test.runner = Microsoft.Testing.Platform global.json config, the output of dotnet test is like this, with exit code 0:
Determining projects to restore...
Restored /Users/patrick/Documents/GitHub/repro2/test.fsproj (in 610 ms).
test -> /Users/patrick/Documents/GitHub/repro2/bin/Debug/net10.0/test.dll
Run tests: '/Users/patrick/Documents/GitHub/repro2/bin/Debug/net10.0/test.dll' [net10.0|arm64]
Tests succeeded: '/Users/patrick/Documents/GitHub/repro2/bin/Debug/net10.0/test.dll' [net10.0|arm64]
If you don't already know that an actually-running test project will output this additional line:
Passed! - Failed: 0, Passed: 1, Skipped: 0, Total: 1, Duration: 125ms - test.dll (net10.0|arm64)
then there is nothing to tell you that you have configured your project in such a way that no tests will ever run.
(Also ideally one would not have to read the output of dotnet test with your human eyes to know whether your tests ran.)
This is a user error - following the docs correctly will result in a project that works - but it's a pretty hostile failure mode, and the console output "Tests succeeded" is actively a falsehood (no tests in fact succeeded).
Steps To Reproduce
See https://github.com/Smaug123/MicrosoftTestingPlatformWeirdness/ at commit 548b60721f3e4b452b6277b6ccbeb45c1de260ba for a repro. That repo also omits <OutputType>Exe</OutputType> but the result is the same with that included.
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net9.0</TargetFramework>
<IsPackable>false</IsPackable>
<IsTestProject>true</IsTestProject>
<OutputType>Exe</OutputType>
<TestingPlatformDotnetTestSupport>true</TestingPlatformDotnetTestSupport>
</PropertyGroup>
<ItemGroup>
<Compile Include="Test.fs" />
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="18.0.1" />
<PackageReference Include="NUnit" Version="4.3.2" />
<PackageReference Include="NUnit3TestAdapter" Version="5.2.0" />
</ItemGroup>
</Project>
Expected behavior
Either the test correctly fails; or the test runner correctly fails and informs me that there were no tests, ideally telling me why; or the build fails, informing me that no tests will ever run.
Actual behavior
Determining projects to restore...
Restored /Users/patrick/Documents/GitHub/repro2/test.fsproj (in 610 ms).
test -> /Users/patrick/Documents/GitHub/repro2/bin/Debug/net10.0/test.dll
Run tests: '/Users/patrick/Documents/GitHub/repro2/bin/Debug/net10.0/test.dll' [net10.0|arm64]
Tests succeeded: '/Users/patrick/Documents/GitHub/repro2/bin/Debug/net10.0/test.dll' [net10.0|arm64]
Additional context
The answer might be "we're going to deprecate VSTest mode", which is fine, although the error message even in the dotnet test-MTP-mode case is pretty obscure, being this plus a link to a page of documentation that doesn't contain information about how to configure the project correctly:
global.json defines test runner to be Microsoft.Testing.Platform. All projects must use that test runner.
The following test projects are using VSTest test runner:
test.fsproj
I don't know how the testing platform works well enough to say what that message ought to be.
Disabling TestingPlatformDotnetTestSupport will make it work correctly, because we then use vstest and execute the tests. Checking if this should be fixed by adding error for the incomplete setup, or if that is inherent problem that will happen when the child process will return exit 0 without doing anything (and hence we need some kind of "heartbeat").
Yep, I think that screenshot is consistent with my understanding, thanks. The fix in the other direction is to set up the global.json's .test.runner = Microsoft.Testing.Platform to fully set up MTP.