dotnet test swallows unit tests

[livarcocc]

From @DerAlbertCom on Friday, August 9, 2019 12:35:27 PM

dotnet swallows silently xunit tests (maybe also test from other test frameworks) when there is a problem on the test discovery. I filed the same under: https://github.com/xunit/visualstudio.xunit/issues/164

Steps to reproduce

Checkout the repository https://github.com/DerAlbertCom/MisbehavingUnittestRunners run dotnet test. More details why the problem occurs is also there.

Expected behavior

At least the exception within the test discovery shows up. It would be better if it is possible to give more detail on exact type or test method.

Actual behavior

In the sample repository are 6 Unit Test, 2 of them get swallowed without any hint because of an exception within the test discoery on one test.

Environment data

.NET Core SDK (reflecting any global.json):                                                                              Version:   2.2.301
 Commit:    70d6be0814

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.18362
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\2.2.301\

Host (useful for support):
  Version: 3.0.0-preview7-27912-14
  Commit:  4da6ee6450

.NET Core SDKs installed:
  2.1.508 [C:\Program Files\dotnet\sdk]
  2.1.701 [C:\Program Files\dotnet\sdk]
  2.1.801 [C:\Program Files\dotnet\sdk]
  2.2.108 [C:\Program Files\dotnet\sdk]
  2.2.204 [C:\Program Files\dotnet\sdk]
  2.2.301 [C:\Program Files\dotnet\sdk]
  2.2.401 [C:\Program Files\dotnet\sdk]
  3.0.100-preview7-012821 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.0.0-preview7.19365.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.0.0-preview7-27912-14 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.0.0-preview7-27912-14 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]    
To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

Copied from original issue: dotnet/cli#12187

[ShreyasRmsft]

@livarcocc could you run this with diag logs enabled and share them here?

[livarcocc]

@ShreyasRmsft I just transferred the issue here.

@DerAlbertCom can you share diag logs here so that this can be further investigated?

[DerAlbertCom]

@ShreyasRmsft @livarcocc you now that there is a repository with an easy reproduction sample, and a lot more details https://github.com/DerAlbertCom/MisbehavingUnittestRunners

But anyway :) here the logs.

diag.host.19-08-12_19-59-49_08946_6.log diag.log

And yes, the missing tests (Test2 and Test3) are also not in that logs.

[ShreyasRmsft]

@DerAlbertCom i took a quick look. The host diag log that you see is basically open for the test adapters to use as they please. The test host process simply invokes the adapter and listens to exceptions if any. In this case the xunit adapter did not throw any exceptions.

I know for a fact that mstest does log the reason/exception if any when discovery has issues. I would recommend you either try and debug through the xunit adapter to see why there are no logs.

I've seen NUnit also printing messages like failed to discover tests due to failure in loading assembly etc. The mechanism for logging is present, in this case i think the xunit adapter is not leveraging it.

[DerAlbertCom]

@ShreyasRmsft thank for the information.

I filed already the bugs under https://github.com/xunit/visualstudio.xunit/issues/164 and https://github.com/xunit/visualstudio.xunit/issues/165 no reaction so far.

[ShreyasRmsft]

@DerAlbertCom since all of these are open source you should be able to just add a throw in the discovery code of the xunit adapter and see who is handling/catching it.