Upgrading Microsoft.NET.Test.Sdk from 16.9.4 to 16.10.0: tests are ignored silently

[kiwimato]

Description

We have a pipeline in Azure DevOps which stopped running tests after we upgraded Microsoft.NET.Test.Sdk to 16.10.0. Last running version seems to be: 16.9.4

Steps to reproduce

actual dotnet test command:

/opt/hostedtoolcache/dotnet/dotnet test /home/vsts/work/1/s/tests/Something.Tests/Something.Tests.csproj --logger trx --results-directory /home/vsts/work/_temp --no-build -c Release -s /home/vsts/work/1/s/CodeCoverage.runsettings /p:CollectCoverage=true /p:CoverletOutputFormat=opencover -- RunConfiguration.DisableAppDomain=true

Note: I noticed that running the command with --collect:"XPlat Code Coverage" and removing /p:CollectCoverage=true /p:CoverletOutputFormat="opencover" seems to work, but I need the latter since I need the reports.

Project file:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net5.0</TargetFramework>
    <IsPackable>false</IsPackable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="FluentAssertions" Version="5.10.3" />
    <PackageReference Include="NSubstitute" Version="4.2.2" />
    <PackageReference Include="NSubstitute.Analyzers.CSharp" Version="1.0.14" />
    <PackageReference Include="Something.Infrastructure.TestDoubles" Version="1.6.0" />
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.9.4" />
    <PackageReference Include="Moq" Version="4.16.1" />
    <PackageReference Include="xunit" Version="2.4.1" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.4.3">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
    <PackageReference Include="coverlet.collector" Version="3.1.0">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
  </ItemGroup>

  <ItemGroup>
    <ProjectReference Include="..\..\src\Something.Service\Something.Service.csproj" />
  </ItemGroup>

</Project>

Expected behavior

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Data collector 'Code Coverage' message: No code coverage data available. Code coverage is currently supported only on Windows..
Results File: /home/vsts/work/_temp/_fv-az230-957_2021-07-30_13_20_46.trx

Passed!  - Failed:     0, Passed:     8, Skipped:     0, Total:     8, Duration: 1 s - /home/vsts/work/1/s/tests/Something.Tests/bin/Release/net5.0/Something.Tests.dll (net5.0)

Actual behavior

The annoying thing is that it fails silently and we actually noticed this by mistake, we thought everything is fine since the job finishes successfully and without any errors.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.

Attachments:
  /home/vsts/work/_temp/00aeb2b6-5cc0-4b06-8c7f-334f2b3a64fb/coverage.cobertura.xml

Environment

Please share additional details about the test environment.

• Operating system Azure Pipelines: ubuntu-18.04
• Build version of vstest.console Microsoft (R) Test Execution Command Line Tool Version 16.8.0
• Package version of MSTest framework and adapter
• Other installed packages and their versions on the test project

[danielchalmers]

Got around this for a bit by downgrading MSTest* to 2.2.3 but it stopped working again recently.

At one point I randomly removed <Deterministic>false</Deterministic> which fixed it briefly. Upgrading the packages broke it again and going back doesn't fix it anymore.

Visual Studio 2022 Preview 3.1 targeting dotnet 6 on preview 7.

[jakubch1]

@kiwimato could you please paste your CodeCoverage.runsettings ?

[kiwimato]

@jakubch1 sure:

<?xml version="1.0" encoding="utf-8"?>
<!-- File name extension must be .runsettings -->
<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
        <Configuration>
          <CodeCoverage>
            <ModulePaths>
              <Include>
                <!-- Auto detect -->
              </Include>
              <Exclude>
                <!-- Exclude nothing -->
              </Exclude>
            </ModulePaths>
          </CodeCoverage>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
</RunSettings>

[jakubch1]

@kiwimato were you getting any code coverage reports on 16.9.4 ?

[kiwimato]

@jakubch1 yes, actually downgrading Microsoft.NET.Test.Sdk to 16.9.4 made code coverage work again.

[jakubch1]

Could you please paste output of command when using 16.9.4? Is the report opencover?

[kiwimato]

This is an example of working output. And yes it's an opencover report.

/opt/hostedtoolcache/dotnet/dotnet test /home/vsts/work/1/s/tests/Redacted.Tests/Redacted.Infrastructure.Tests.csproj --logger trx --results-directory /home/vsts/work/_temp --no-build -c Release -s /home/vsts/work/1/s/CodeCoverage.runsettings --collect:XPlat Code Coverage /p:CollectCoverage=true /p:CoverletOutputFormat=opencover -- RunConfiguration.DisableAppDomain=true
Test run for /home/vsts/work/1/s/tests/Redacted.Tests/bin/Release/net5.0/Redacted.Infrastructure.Tests.dll (.NETCoreApp,Version=v5.0)
Microsoft (R) Test Execution Command Line Tool Version 16.8.0
Copyright (c) Microsoft Corporation.  All rights reserved.
Data collector 'Code Coverage' message: No code coverage data available. Code coverage is currently supported only on Windows..
Data collection : Unable to find a datacollector with friendly name 'XPlat Code Coverage'.
Data collection : Could not find data collector 'XPlat Code Coverage'
Results File: /home/vsts/work/_temp/_fv-az101-289_2021-08-18_15_10_25.trx

Passed!  - Failed:     0, Passed:     1, Skipped:     0, Total:     1, Duration: 21 ms - /home/vsts/work/1/s/tests/Redacted.Tests/bin/Release/net5.0/Redacted.Infrastructure.Tests.dll (net5.0)

Calculating coverage result...
  Generating report '/home/vsts/work/1/s/tests/Redacted.Tests/coverage.opencover.xml'

+----------------------------------------+-------+--------+--------+
| Module                                 | Line  | Branch | Method |
+----------------------------------------+-------+--------+--------+
| Redacted.Api            | 0%    | 0%     | 0%     |
+----------------------------------------+-------+--------+--------+
| Redacted.Domain         | 0%    | 100%   | 0%     |
+----------------------------------------+-------+--------+--------+
| Redacted.Infrastructure | 1.98% | 100%   | 7.14%  |
+----------------------------------------+-------+--------+--------+

+---------+-------+--------+--------+
|         | Line  | Branch | Method |
+---------+-------+--------+--------+
| Total   | 0.45% | 0%     | 1.47%  |
+---------+-------+--------+--------+
| Average | 0.66% | 66.66% | 2.38%  |
+---------+-------+--------+--------+

[jakubch1]

Thanks for info. In 16.10.0 we added support for dynamic code coverage in Linux. Unfortunately there is a bug in old version of TestPlatform which is causing this behaviour.

You have 2 options:

1. Switch to dynamic code coverage. To do that you need to upgrade your dotnet version to latest stable and disable opencover (remove: /p:CollectCoverage=true /p:CoverletOutputFormat=opencover). This should give you better AzDo integration. If you upload your trx file to AzDo you will see coverage number in AzDo and even diff code coverage for PRs.
2. Continue using OpenCover and disable dynamic code coverage. To do that just remove your runsettings (your runsettings are enabling dynamic code coverage)

[Evangelink]

@jakubch1 @MarcoRossignoli Is this issue specifically related to mstest? It seems to be more of a Test Platform issue to me (in which case I will transfer it). Also, I think it can be closed.

[MarcoRossignoli]

@kiwimato is this issue still present?

[Evangelink]

Closing this issue. Feel free to ping if you have any other problem.

Thank you.