dotnet test and VsTest swallows xUnit tests

[DerAlbertCom]

copied from https://github.com/xunit/visualstudio.xunit/issues/164 and https://github.com/xunit/visualstudio.xunit/issues/165.

xunit swallows silently xunit tests (maybe also test from other test frameworks) when there is a problem on the test discovery. I filed the same thing in the dotnet cli repo: https://github.com/dotnet/cli/issues/12187

Steps to reproduce

Checkout the repository https://github.com/DerAlbertCom/MisbehavingUnittestRunners run dotnet test. More details why the problem occurs is also there.

Expected behavior

At least the exception within the test discovery shows up. It would be better if it is possible to give more detail on exact type or test method.

Actual behavior of dotnet test

In the sample repository are 6 Unit Test, 2 of them get swallowed without any hint because of an exception within the test discovery on one test.

Actual behavior of Visual Studio

Test withing Visual Studio 2019 16.2 . In the sample repository are 6 Unit Test, 2 of are not run without any hint because of an exception within the test discovery on one test.

Environment data

.NET Core SDK (reflecting any global.json):                                                                              Version:   2.2.301
 Commit:    70d6be0814

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.18362
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\2.2.301\

Host (useful for support):
  Version: 3.0.0-preview7-27912-14
  Commit:  4da6ee6450

.NET Core SDKs installed:
  2.1.508 [C:\Program Files\dotnet\sdk]
  2.1.701 [C:\Program Files\dotnet\sdk]
  2.1.801 [C:\Program Files\dotnet\sdk]
  2.2.108 [C:\Program Files\dotnet\sdk]
  2.2.204 [C:\Program Files\dotnet\sdk]
  2.2.301 [C:\Program Files\dotnet\sdk]
  2.2.401 [C:\Program Files\dotnet\sdk]
  3.0.100-preview7-012821 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.0.0-preview7.19365.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.0.0-preview7-27912-14 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.0.0-preview7-27912-14 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]    
To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

[DerAlbertCom]

In microsoft/vstest#2123 is mentioned that the test adapter seems to capture the exception during discovery, because of that the Test2 and Test3 are not run and lost.

I took a look at the SourceCode of the VsTestRunner.

The Exception Handling while https://github.com/xunit/visualstudio.xunit/blob/8b6ddc907a600dcad3e6f961486e54d60283e86d/src/xunit.runner.visualstudio/VsTestRunner.cs#L247

and

https://github.com/xunit/visualstudio.xunit/blob/8b6ddc907a600dcad3e6f961486e54d60283e86d/src/xunit.runner.visualstudio/VsTestRunner.cs#L316

An Exception should be logged.

But on the diag logs of dotnet test the doesn't appear.

Theses are the only logs that the VsTestRunners seems to log to the testrunner from VS and dotnet test.

[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.1 (64-bit .NET Core 4.6.27817.03)
[xUnit.net 00:00:00.33]   Discovering: XUnitTestProject1
[xUnit.net 00:00:00.38]   Discovered:  XUnitTestProject1
[xUnit.net 00:00:00.39]   Starting:    XUnitTestProject1
[xUnit.net 00:00:00.46]     XUnitTestProject1.UnitTest2.Test10 [SKIP]
[xUnit.net 00:00:00.46]       Skipped with Fact
[xUnit.net 00:00:00.47]     XUnitTestProject1.UnitTest1.Test1 [SKIP]
[xUnit.net 00:00:00.47]       Skipped with Fact
[xUnit.net 00:00:00.49]     XUnitTestProject1.UnitTest2.Test20 [SKIP]
[xUnit.net 00:00:00.49]       Skipped
[xUnit.net 00:00:00.49]   Finished:    XUnitTestProject1

So, the Exception or whatever the reason is for not running Test2 and Test3 will be swallowed a level deeper.

[DerAlbertCom]

Any Ideas, About this?

[DerAlbertCom]

ReSharper relies also on the results of xunit and behaves not well because of that, https://youtrack.jetbrains.com/issue/RSRP-475867