coverlet icon indicating copy to clipboard operation
coverlet copied to clipboard

Collecting code coverage prevents all test discovery on net472 only

Open AArnott opened this issue 6 years ago • 24 comments

This repros locally and in Azure Pipelines builds.

git clone https://github.com/AArnott/Xunit.StaFact.git
cd Xunit.StaFact\src
git checkout 6d1a5e6023f612ad00d2e00f5204f362dc0a71ce
dotnet test --filter "TestCategory!=FailureExpected" -v n /p:CollectCoverage=true

Notice how net472 runs 0 tests and reports:

+---------------+------+--------+--------+
| Module        | Line | Branch | Method |
+---------------+------+--------+--------+
| Xunit.StaFact | 0%   | 0%     | 0%     |
+---------------+------+--------+--------+

+---------+------+--------+--------+
|         | Line | Branch | Method |
+---------+------+--------+--------+
| Total   | 0%   | 0%     | 0%     |
+---------+------+--------+--------+
| Average | 0%   | 0%     | 0%     |
+---------+------+--------+--------+

While the netcoreapp targets report non-zero results:

+---------------+--------+--------+--------+
| Module        | Line   | Branch | Method |
+---------------+--------+--------+--------+
| Xunit.StaFact | 86.44% | 62.19% | 78.33% |
+---------------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 86.44% | 62.19% | 78.33% |
+---------+--------+--------+--------+
| Average | 86.44% | 62.19% | 78.33% |
+---------+--------+--------+--------+

In fact simply comparing the output of these two commands shows that the mere collection of code coverage is what blocks test discovery:

 dotnet test --filter "TestCategory!=FailureExpected" /p:CollectCoverage=true -f net472 
 dotnet test --filter "TestCategory!=FailureExpected" -f net472 

AArnott avatar Oct 04 '19 17:10 AArnott

@AArnott I don't see any issue on coverlet side for netfx...

@bradwilson @onovotny sorry for the inconvenience....I need to attach and debug the reason why on netfx+coverlet lead xunit to fail the discovering of tests(weird no issue with netcoreapp)...is there a way(guide/docs) on how attach a debugger to dotnet test? Tried with set VSTEST_HOST_DEBUG=1 but no luck with symbols.

MarcoRossignoli avatar Oct 05 '19 07:10 MarcoRossignoli

@MarcoRossignoli The way I do it is add Debugger.Launch() to some early executing code in my assembly. I tried that here, in my own test discoverer class, but it was never invoked, suggesting the problem is earlier than my code is loaded.

AArnott avatar Oct 05 '19 12:10 AArnott

@AArnott not sure if it's related but if I change the order of test the behaviour is different...I wonder if the issue is how custom attribute interact with xunit for net472 context. Try this:

  1. run as is
Test run for C:\git\coverletissue\Xunit.StaFact\bin\Xunit.StaFact.Tests\Debug\net472\Xunit.StaFact.Tests.dll(.NETFramework,Version=v4.7.2)
Microsoft (R) Test Execution Command Line Tool Version 16.3.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.1 (32-bit Desktop .NET 4.0.30319.42000)
[xUnit.net 00:00:00.64]   Discovering: Xunit.StaFact.Tests
[xUnit.net 00:00:00.86]   Discovered:  Xunit.StaFact.Tests
No test matches the given testcase filter `TestCategory!=FailureExpected` in C:\git\coverletissue\Xunit.StaFact\bin\Xunit.StaFact.Tests\Debug\net472\Xunit.StaFact.Tests.dll


Calculating coverage result...
  Generating report 'C:\git\coverletissue\Xunit.StaFact\bin\Xunit.StaFact.Tests\Debug\net472\coverage.cobertura.xml'

+---------------+------+--------+--------+
| Module        | Line | Branch | Method |
+---------------+------+--------+--------+
| Xunit.StaFact | 0%   | 0%     | 0%     |
+---------------+------+--------+--------+

+---------+------+--------+--------+
|         | Line | Branch | Method |
+---------+------+--------+--------+
| Total   | 0%   | 0%     | 0%     |
+---------+------+--------+--------+
| Average | 0%   | 0%     | 0%     |
+---------+------+--------+--------+

     3>Done Building Project "C:\git\coverletissue\Xunit.StaFact\src\Xunit.StaFact.Tests\Xunit.StaFact.Tests.csproj" (VSTest target(s)).
     1>Done Building Project "C:\git\coverletissue\Xunit.StaFact\src\Xunit.StaFact.sln" (VSTest target(s)).

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:06.10

C:\git\coverletissue\Xunit.StaFact\src (HEAD detached at 6d1a5e6 -> origin)
λ
  1. Move Fact_OnMTAThread in file Sample.cs as FIRST test and re-run

public class Samples
{
    /// <summary>
    /// Demonstrates that xunit [Fact] behavior is to invoke the test on an MTA thread.
    /// </summary>
    [Fact]
    public async Task Fact_OnMTAThread()
    {
        var expectedApartment = RuntimeInformation.IsOSPlatform(OSPlatform.Windows) ? ApartmentState.MTA : ApartmentState.Unknown;
        Assert.Equal(expectedApartment, Thread.CurrentThread.GetApartmentState());
        await Task.Yield();
        Assert.Equal(expectedApartment, Thread.CurrentThread.GetApartmentState());
    }

    /// <summary>
    /// Demonstrates that the <see cref="UIFactAttribute"/> can run on all platforms,
    /// and emulates a UI thread (not specific to WPF or WinForms) with a <see cref="SynchronizationContext"/>
    /// that keeps yielding awaits on the original thread.
    /// </summary>
    [UIFact]
    public async Task UIFact_OnSTAThread()
    {
        int initialThread = Environment.CurrentManagedThreadId;
        Assert.NotNull(SynchronizationContext.Current);
        var expectedApartment = RuntimeInformation.IsOSPlatform(OSPlatform.Windows) ? ApartmentState.STA : ApartmentState.Unknown;
        Assert.Equal(expectedApartment, Thread.CurrentThread.GetApartmentState());

        await Task.Yield();
        Assert.Equal(initialThread, Environment.CurrentManagedThreadId);
        Assert.NotNull(SynchronizationContext.Current);
    }

Coverage is the same but one test is found...I wonder if something goes wrong and process is killed or dead, this could explain empty collected file

Test run for C:\git\coverletissue\Xunit.StaFact\bin\Xunit.StaFact.Tests\Debug\net472\Xunit.StaFact.Tests.dll(.NETFramework,Version=v4.7.2)                   
Microsoft (R) Test Execution Command Line Tool Version 16.3.0                                                                                                
Copyright (c) Microsoft Corporation.  All rights reserved.                                                                                                   
                                                                                                                                                             
Starting test execution, please wait...                                                                                                                      
                                                                                                                                                             
A total of 1 test files matched the specified pattern.                                                                                                       
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.1 (32-bit Desktop .NET 4.0.30319.42000)                                                                
[xUnit.net 00:00:00.63]   Discovering: Xunit.StaFact.Tests                                                                                                   
[xUnit.net 00:00:00.88]   Discovered:  Xunit.StaFact.Tests                                                                                                   
[xUnit.net 00:00:00.89]   Starting:    Xunit.StaFact.Tests                                                                                                   
[xUnit.net 00:00:01.09]   Finished:    Xunit.StaFact.Tests                                                                                                   
  V Samples.Fact_OnMTAThread [57ms]                                                                                                                          
                                                                                                                                                             
Test Run Successful.                                                                                                                                         
Total tests: 1                                                                                                                                               
     Passed: 1                                                                                                                                               
 Total time: 2,1416 Seconds                                                                                                                                  
                                                                                                                                                             
Calculating coverage result...                                                                                                                               
  Generating report 'C:\git\coverletissue\Xunit.StaFact\bin\Xunit.StaFact.Tests\Debug\net472\coverage.cobertura.xml'                                         
                                                                                                                                                             
+---------------+------+--------+--------+                                                                                                                   
| Module        | Line | Branch | Method |                                                                                                                   
+---------------+------+--------+--------+                                                                                                                   
| Xunit.StaFact | 0%   | 0%     | 0%     |                                                                                                                   
+---------------+------+--------+--------+                                                                                                                   
                                                                                                                                                             
+---------+------+--------+--------+                                                                                                                         
|         | Line | Branch | Method |                                                                                                                         
+---------+------+--------+--------+                                                                                                                         
| Total   | 0%   | 0%     | 0%     |                                                                                                                         
+---------+------+--------+--------+                                                                                                                         
| Average | 0%   | 0%     | 0%     |                                                                                                                         
+---------+------+--------+--------+                                                                                                                         
                                                                                                                                                             
     3>Done Building Project "C:\git\coverletissue\Xunit.StaFact\src\Xunit.StaFact.Tests\Xunit.StaFact.Tests.csproj" (VSTest target(s)).                     
     1>Done Building Project "C:\git\coverletissue\Xunit.StaFact\src\Xunit.StaFact.sln" (VSTest target(s)).                                                  
                                                                                                                                                             
Build succeeded.                                                                                                                                             
    0 Warning(s)                                                                                                                                             
    0 Error(s)                                                                                                                                               
                                                                                                                                                             
Time Elapsed 00:00:06.33                                                                                                                                     
                                                                                                                                                             
C:\git\coverletissue\Xunit.StaFact\src (HEAD detached at 6d1a5e6 -> origin)                                                                                  
λ                                                                                                                                                            

MarcoRossignoli avatar Oct 07 '19 07:10 MarcoRossignoli

If you run with test host log enabled you can also some difference for the two execution

dotnet test --filter "TestCategory!=FailureExpected"  /p:CollectCoverage=true --diag:log.txt

No test run

TpTrace Verbose: 0 : 25876, 9, 2019/10/07, 09:51:26.919, 10232781449091, testhost.x86.exe, TcpClientExtensions.MessageLoopAsync: Polling on remoteEndPoint: 127.0.0.1:62192 localEndPoint: 127.0.0.1:62193
TpTrace Information: 0 : 25876, 12, 2019/10/07, 09:51:27.009, 10232782352289, testhost.x86.exe, [xUnit.net 00:00:00.67]   Discovering: Xunit.StaFact.Tests
TpTrace Verbose: 0 : 25876, 12, 2019/10/07, 09:51:27.009, 10232782353896, testhost.x86.exe, TestRequestHandler.SendData:  sending data from testhost: {"Version":2,"MessageType":"TestSession.Message","Payload":{"MessageLevel":0,"Message":"[xUnit.net 00:00:00.67]   Discovering: Xunit.StaFact.Tests"}}
TpTrace Information: 0 : 25876, 12, 2019/10/07, 09:51:27.230, 10232784566633, testhost.x86.exe, [xUnit.net 00:00:00.89]   Discovered:  Xunit.StaFact.Tests
TpTrace Verbose: 0 : 25876, 12, 2019/10/07, 09:51:27.231, 10232784568426, testhost.x86.exe, TestRequestHandler.SendData:  sending data from testhost: {"Version":2,"MessageType":"TestSession.Message","Payload":{"MessageLevel":0,"Message":"[xUnit.net 00:00:00.89]   Discovered:  Xunit.StaFact.Tests"}}
TpTrace Verbose: 0 : 25876, 3, 2019/10/07, 09:51:27.258, 10232784840588, testhost.x86.exe, BaseRunTests.RunTestInternalWithExecutors: Completed running tests for executor://xunit/VsTestRunner2/net
TpTrace Warning: 0 : 25876, 3, 2019/10/07, 09:51:27.260, 10232784857803, testhost.x86.exe, No test matches the given testcase filter `TestCategory!=FailureExpected` in C:\git\coverletissue\Xunit.StaFact\bin\Xunit.StaFact.Tests\Debug\net472\Xunit.StaFact.Tests.dll
TpTrace Verbose: 0 : 25876, 3, 2019/10/07, 09:51:27.260, 10232784859341, testhost.x86.exe, TestRequestHandler.SendData:  sending data from testhost: {"Version":2,"MessageType":"TestSession.Message","Payload":{"MessageLevel":1,"Message":"No test matches the given testcase filter `TestCategory!=FailureExpected` in C:\\git\\coverletissue\\Xunit.StaFact\\bin\\Xunit.StaFact.Tests\\Debug\\net472\\Xunit.StaFact.Tests.dll"}}
TpTrace Information: 0 : 25876, 3, 2019/10/07, 09:51:27.264, 10232784905541, testhost.x86.exe, Sending test run complete
TpTrace Verbose: 0 : 25876, 3, 2019/10/07, 09:51:27.291, 10232785174237, testhost.x86.exe, TestRequestHandler.SendData:  sending data from testhost: {"Version":2,"MessageType":"TestExecution.Completed","Payload":{"TestRunCompleteArgs":{"TestRunStatistics":{"ExecutedTests":0,"Stats":{}},"IsCanceled":false,"IsAborted":false,"Error":null,"AttachmentSets":[],"ElapsedTimeInRunningTests":"00:00:00.9380405","Metrics":{}},"LastRunTests":{"NewTestResults":[],"TestRunStatistics":{"ExecutedTests":0,"Stats":{}},"ActiveTests":[]},"RunAttachments":[],"ExecutorUris":[]}}
TpTrace Verbose: 0 : 25876, 3, 2019/10/07, 09:51:27.292, 10232785178258, testhost.x86.exe, BaseRunTests.RunTests: Run is complete.

One test run

TpTrace Verbose: 0 : 29100, 8, 2019/10/07, 09:47:02.697, 10230139228025, testhost.x86.exe, TcpClientExtensions.MessageLoopAsync: Polling on remoteEndPoint: 127.0.0.1:62093 localEndPoint: 127.0.0.1:62094
TpTrace Information: 0 : 29100, 12, 2019/10/07, 09:47:02.904, 10230141301529, testhost.x86.exe, [xUnit.net 00:00:00.73]   Discovering: Xunit.StaFact.Tests
TpTrace Verbose: 0 : 29100, 12, 2019/10/07, 09:47:02.904, 10230141303138, testhost.x86.exe, TestRequestHandler.SendData:  sending data from testhost: {"Version":2,"MessageType":"TestSession.Message","Payload":{"MessageLevel":0,"Message":"[xUnit.net 00:00:00.73]   Discovering: Xunit.StaFact.Tests"}}
TpTrace Information: 0 : 29100, 12, 2019/10/07, 09:47:03.155, 10230143808181, testhost.x86.exe, [xUnit.net 00:00:00.98]   Discovered:  Xunit.StaFact.Tests
TpTrace Verbose: 0 : 29100, 12, 2019/10/07, 09:47:03.155, 10230143809631, testhost.x86.exe, TestRequestHandler.SendData:  sending data from testhost: {"Version":2,"MessageType":"TestSession.Message","Payload":{"MessageLevel":0,"Message":"[xUnit.net 00:00:00.98]   Discovered:  Xunit.StaFact.Tests"}}
TpTrace Information: 0 : 29100, 12, 2019/10/07, 09:47:03.162, 10230143883820, testhost.x86.exe, No tests matched the filter because it contains one or more properties that are not valid (TestCategory). Specify filter expression containing valid properties (DisplayName, FullyQualifiedName).
TpTrace Information: 0 : 29100, 12, 2019/10/07, 09:47:03.167, 10230143931897, testhost.x86.exe, [xUnit.net 00:00:00.99]   Starting:    Xunit.StaFact.Tests
TpTrace Verbose: 0 : 29100, 12, 2019/10/07, 09:47:03.167, 10230143933047, testhost.x86.exe, TestRequestHandler.SendData:  sending data from testhost: {"Version":2,"MessageType":"TestSession.Message","Payload":{"MessageLevel":0,"Message":"[xUnit.net 00:00:00.99]   Starting:    Xunit.StaFact.Tests"}}
TpTrace Information: 0 : 29100, 12, 2019/10/07, 09:47:03.171, 10230143972693, testhost.x86.exe, AssemblyResolver.OnResolve: xunit.execution.desktop, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c: Resolving assembly.
TpTrace Information: 0 : 29100, 12, 2019/10/07, 09:47:03.173, 10230143990590, testhost.x86.exe, AssemblyResolver.OnResolve: Resolved assembly: xunit.execution.desktop, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c, from path: C:\git\coverletissue\Xunit.StaFact\bin\Xunit.StaFact.Tests\Debug\net472\xunit.execution.desktop.dll
TpTrace Information: 0 : 29100, 12, 2019/10/07, 09:47:03.173, 10230143993405, testhost.x86.exe, AssemblyResolver.OnResolve: xunit.execution.desktop, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c: Resolving assembly.
TpTrace Information: 0 : 29100, 12, 2019/10/07, 09:47:03.173, 10230143994293, testhost.x86.exe, AssemblyResolver.OnResolve: xunit.execution.desktop, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c: Resolved from cache.
TpTrace Information: 0 : 29100, 12, 2019/10/07, 09:47:03.174, 10230143998179, testhost.x86.exe, AssemblyResolver.OnResolve: xunit.abstractions, Version=2.0.0.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c: Resolving assembly.
TpTrace Information: 0 : 29100, 12, 2019/10/07, 09:47:03.174, 10230143999100, testhost.x86.exe, AssemblyResolver.OnResolve: xunit.abstractions, Version=2.0.0.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c: Resolved from cache.
TpTrace Information: 0 : 29100, 12, 2019/10/07, 09:47:03.174, 10230144003206, testhost.x86.exe, AssemblyResolver.OnResolve: xunit.core, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c: Resolving assembly.
TpTrace Information: 0 : 29100, 12, 2019/10/07, 09:47:03.175, 10230144017437, testhost.x86.exe, AssemblyResolver.OnResolve: Resolved assembly: xunit.core, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c, from path: C:\git\coverletissue\Xunit.StaFact\bin\Xunit.StaFact.Tests\Debug\net472\xunit.core.dll
TpTrace Information: 0 : 29100, 12, 2019/10/07, 09:47:03.176, 10230144022986, testhost.x86.exe, AssemblyResolver.OnResolve: xunit.abstractions, Version=2.0.0.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c: Resolving assembly.
TpTrace Information: 0 : 29100, 12, 2019/10/07, 09:47:03.176, 10230144023871, testhost.x86.exe, AssemblyResolver.OnResolve: xunit.abstractions, Version=2.0.0.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c: Resolved from cache.
TpTrace Verbose: 0 : 29100, 15, 2019/10/07, 09:47:03.272, 10230144982900, testhost.x86.exe, TestExecutionRecorder.RecordStart: Starting test: Samples.Fact_OnMTAThread.
TpTrace Verbose: 0 : 29100, 15, 2019/10/07, 09:47:03.364, 10230145899927, testhost.x86.exe, TestExecutionRecorder.RecordResult: Received result for test: Samples.Fact_OnMTAThread.
TpTrace Verbose: 0 : 29100, 15, 2019/10/07, 09:47:03.369, 10230145952040, testhost.x86.exe, TestExecutionRecorder.RecordEnd: test: Samples.Fact_OnMTAThread execution completed.
TpTrace Warning: 0 : 29100, 15, 2019/10/07, 09:47:03.369, 10230145953150, testhost.x86.exe, InProgressTests is null
TpTrace Information: 0 : 29100, 12, 2019/10/07, 09:47:03.372, 10230145983163, testhost.x86.exe, [xUnit.net 00:00:01.20]   Finished:    Xunit.StaFact.Tests
TpTrace Verbose: 0 : 29100, 12, 2019/10/07, 09:47:03.372, 10230145984358, testhost.x86.exe, TestRequestHandler.SendData:  sending data from testhost: {"Version":2,"MessageType":"TestSession.Message","Payload":{"MessageLevel":0,"Message":"[xUnit.net 00:00:01.20]   Finished:    Xunit.StaFact.Tests"}}
TpTrace Verbose: 0 : 29100, 3, 2019/10/07, 09:47:03.407, 10230146335086, testhost.x86.exe, BaseRunTests.RunTestInternalWithExecutors: Completed running tests for executor://xunit/VsTestRunner2/net
TpTrace Information: 0 : 29100, 3, 2019/10/07, 09:47:03.412, 10230146378853, testhost.x86.exe, Sending test run complete

MarcoRossignoli avatar Oct 07 '19 08:10 MarcoRossignoli

@AArnott any news?

MarcoRossignoli avatar Jan 02 '20 23:01 MarcoRossignoli

I didn't realize you were waiting on me. It looked like you were calling out results of your investigation as you went. What would you like me to do to help? Reordering test methods isn't an acceptable workaround IMO. I mean, sure I could do that, but isn't this a general problem?

AArnott avatar Jan 02 '20 23:01 AArnott

Reordering test methods isn't an acceptable workaround IMO

Sure I'm not saying that, but I wonder if this strange behaviour ring a bell to you to help to understand better, coverlet only instrument asm adding some "register hit" static method on line and branches. BTW I'll try to debug xunit adapter where the test are discovered.

MarcoRossignoli avatar Jan 03 '20 10:01 MarcoRossignoli

OK, thanks. Ya, none of what you've discovered causes any root cause ideas to come to mind.

AArnott avatar Jan 03 '20 14:01 AArnott

Ok ok no prob, thank's.

MarcoRossignoli avatar Jan 03 '20 14:01 MarcoRossignoli

I'm getting similar results with this project: https://github.com/saucecontrol/Blake2Fast/tree/azure-coverage

What I've found is that when I run locally, I get correct results with netfx target on 32-bit

dotnet test tests\Blake2.Test -c Coverage -s tests\runsettings --logger trx

with the following runsettings:

<RunSettings>
    <RunConfiguration>
        <TargetPlatform>x86</TargetPlatform>
        <DisableAppDomain>true</DisableAppDomain>
    </RunConfiguration>
</RunSettings>

However, if I switch to <TargetPlatform>x64</TargetPlatform>, I get 0% coverage.

On Azure Pipelines, I get 0% coverage under either config.

The netcore targets are correct no matter how I run.

saucecontrol avatar Jan 25 '20 04:01 saucecontrol

Thanks for the info, I'm on it at the moment maybe it's also related to another similar issue #700

MarcoRossignoli avatar Jan 25 '20 07:01 MarcoRossignoli

Great, thanks! Mine doesn't look the same as #700, and actually it might not be the same as this one either. My tests aren't failing to run, only to generate correct coverage.

netfx 32-bit results:

Test Run Successful.
Total tests: 1032
     Passed: 1032
 Total time: 1.7600 Seconds

Calculating coverage result...
  Generating report 'C:\git\blake2fast\tests\..\out\coverage\opencover.net46.xml'

+-------------------------+--------+--------+--------+
| Module                  | Line   | Branch | Method |
+-------------------------+--------+--------+--------+
| SauceControl.Blake2Fast | 98.54% | 76.66% | 70.58% |
+-------------------------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 98.54% | 76.66% | 70.58% |
+---------+--------+--------+--------+
| Average | 98.54% | 76.66% | 70.58% |
+---------+--------+--------+--------+

netfx 64-bit results

Test Run Successful.
Total tests: 1032
     Passed: 1032
 Total time: 2.0768 Seconds

Calculating coverage result...
  Generating report 'C:\git\blake2fast\tests\..\out\coverage\opencover.net46.xml'

+-------------------------+------+--------+--------+
| Module                  | Line | Branch | Method |
+-------------------------+------+--------+--------+
| SauceControl.Blake2Fast | 0%   | 0%     | 0%     |
+-------------------------+------+--------+--------+

+---------+------+--------+--------+
|         | Line | Branch | Method |
+---------+------+--------+--------+
| Total   | 0%   | 0%     | 0%     |
+---------+------+--------+--------+
| Average | 0%   | 0%     | 0%     |
+---------+------+--------+--------+

This is from a local run. Again, it reports 0% under both configs when I run it in Azure Pipelines.

By the way, I've tried both the default collector and coverlet.collector with the same results. Never could figure out where it's failing, though.

If you get a chance to run my example, be aware that the different targets will have wildly different coverage, by design. The netcore targets have a lot of hardware intrinsics code plus fallback scalar code. The netfx targets run only the scalar code, and the netcore targets run only the intrinsics (which are different between 2.x and 3.x). Of course, that's why I need to run all targets to get a complete coverage report.

saucecontrol avatar Jan 25 '20 07:01 saucecontrol

Ok thank's!

MarcoRossignoli avatar Jan 25 '20 08:01 MarcoRossignoli

@saucecontrol can you try to use collectors? the 0% issue usually is related to know issue with msbuild

https://github.com/tonerdo/coverlet/blob/master/Documentation/KnowIssues.md#1-vstest-stops-process-execution-earlydotnet-test

For now if you use a collector with version greater than 1.0.0(last is 1.2.0) you need also to use preview sdk test plat

https://github.com/tonerdo/coverlet/blob/master/Documentation/KnowIssues.md#2-upgrade-coverletcollector-to-version--100

MarcoRossignoli avatar Jan 25 '20 14:01 MarcoRossignoli

I've tried coverlet.collectors before with no change in behavior, so I rolled back to the simpler config. I have just tried again with the latest version(s), and no change. I updated my azure-coverage branch with the collectors config in case you have time to look it over.

I'm running tests with the following cmdline

dotnet test tests\Blake2.Test -c Coverage -s tests\runsettings.collector --logger trx --collect:"XPlat Code Coverage"

Is there something I'm missing?

Also, I'm happy to create a new issue if this is a different problem. My tests are being discovered fine, it's just the coverage that's wrong. I didn't see any others issues specific to netfx, and my failures have only ever happened on that target.

saucecontrol avatar Jan 25 '20 18:01 saucecontrol

Better create a new issue seem a different case BTW this is our hello word sample https://github.com/tonerdo/coverlet/tree/master/Documentation/Examples/VSTest/HelloWorld

I've tried coverlet.collectors before with no change in behavior, so I rolled back to the simpler config.

And sorry I asked same thing two times did a mess with other issues and on mobile :man_facepalming:

MarcoRossignoli avatar Jan 25 '20 18:01 MarcoRossignoli

Ha, no problem 😄. I appreciate you looking into it.

Created a new issue https://github.com/tonerdo/coverlet/issues/705

saucecontrol avatar Jan 25 '20 18:01 saucecontrol

I am seeing a similar issue. MS's coverage collector in VS enterprise/test sees the coverage just fine, but coverlet is "blind" to all coverage that should be available from a particular XUnit test project. It seems to be linked to a particular call to a dependency lib that starts a thread. I will work to create a simple repro sln.

If anyone else is experiencing this, this link may help to use MS's collector in your pipeline to collect coverage data. This is my plan B. https://medium.com/swlh/generating-code-coverage-reports-in-dotnet-core-bb0c8cca66

jasells avatar Feb 22 '22 22:02 jasells

I will work to create a simple repro sln.

This would be great thx.

MarcoRossignoli avatar Feb 23 '22 08:02 MarcoRossignoli

OK i have a repro here. It may deserve it's own issue, since it is .NetStnd and involves a 3rd party lib: NetMQ. But, the symptoms are the same.

jasells avatar Feb 23 '22 15:02 jasells

OK i have a repro here. It may deserve it's own issue, since it is .NetStnd and involves a 3rd party lib: NetMQ. But, the symptoms are the same.

So, should I open a new issue, or stay on this one? Given the title of this issue, mine is really a different issue, though related as it seems highly likely to have the same root cause...

jasells avatar Feb 25 '22 16:02 jasells

If anyone else is experiencing this, this link may help to use MS's collector in your pipeline to collect coverage data. This is my plan B. https://medium.com/swlh/generating-code-coverage-reports-in-dotnet-core-bb0c8cca66

FYI, I was able to use this post as a go-by for collecting coverage using Microsoft.CodeCoverage tool to collect and then to convert the output to something that the ReportGenerator tool could use to report line % coverage with no code changes to the code under test.

jasells avatar Feb 25 '22 16:02 jasells

So, should I open a new issue, or stay on this one? Given the title of this issue, mine is really a different issue, though related as it seems highly likely to have the same root cause...

Open a new issue pls.

MarcoRossignoli avatar Mar 09 '22 07:03 MarcoRossignoli

will do.

jasells avatar Mar 13 '22 20:03 jasells

The issue might be out-dated because new releases are available.

Please reopen the issue if the bug still exists and can be reproduced with latest preview from coverlet nightly build.

see Consume nightly build

Bertk avatar Jan 22 '24 08:01 Bertk