coverlet Solution wide coverage

Coverlet should be able to get coverage for all test projects in a specified solution file

related issues https://github.com/tonerdo/coverlet/issues/165

Mar 03 '19 15:03 tonerdo

FYI, We use coverlet msbuild tool to run tests for all tests projects in a solution file with the next command:

dotnet test solution.sln /p:CollectCoverage=true /maxcpucount:1

The /maxcpucount:1 argument is necessary to prevent parallelizable code instrumentation. Parallelizable instrumentation lock files and block other instance of coverlet

Apr 17 '19 19:04 pantosha

That'll produce individual test reports, but won't produce a single combined test report (my impression of what this enhancement is for).

Apr 17 '19 19:04 IGx89

@IGx89 if you run

dotnet test solution.sln /p:CollectCoverage=true /p:Exclude="[xunit*]*" /p:CoverletOutput="../coverage.json" /p:MergeWith="../coverage.json" /maxcpucount:1

it will create a single coverage file and will report the complete coverage after each test.

some things to consider with this approach:

You have to make sure to delete the coverage.json file after the run or the next run will re-use it
the path in CoverletOutput and MergeWith are relative to the test project being run

May 08 '19 08:05 joemey

Directory.Build.props:

<PropertyGroup>
    <CoverletOutput>./coverage/coverage.xml</CoverletOutput>
    <CoverletOutputFormat>opencover</CoverletOutputFormat>
</PropertyGroup>

Directory.Build.targets:

<Target Name="CoverRunPreparation" BeforeTargets="GenerateCoverageResult" Condition="'$(CollectCoverage)' == 'true'">
        <ItemGroup>
            <CoverletOutputPath Include="$(CoverletOutput)" />
        </ItemGroup>
        <PropertyGroup>
            <CoverletOutput Condition="'$(CoverletOutput)' != '' AND '$(TargetFrameworks)' != ''">%(CoverletOutputPath.RootDir)/%(CoverletOutputPath.Directory)/%(CoverletOutputPath.Filename).$(TargetFramework)%(CoverletOutputPath.Extension)</CoverletOutput>
            <CoverletOutput Condition="'$(CoverletOutput)' == '' AND '$(TargetFrameworks)' == ''">$(ProjectDir)coverage.xml</CoverletOutput>
            <CoverletOutput Condition="'$(CoverletOutput)' == '' AND '$(TargetFrameworks)' != ''">$(ProjectDir)coverage.$(TargetFramework).xml</CoverletOutput>
        </PropertyGroup>
</Target>

May 22 '19 11:05 viceice

@joemey how does MergeWith work with multiple CoverletOutputFormat?

Jul 16 '19 12:07 iron9light

@joemey how does MergeWith work with multiple CoverletOutputFormat?

@iron9light take a look at https://github.com/tonerdo/coverlet/issues/474

Jul 16 '19 13:07 MarcoRossignoli

I have tried the solution suggested by @joemey and it works, but only for json, but it does not for opencover.

For this command dotnet test /p:CollectCoverage=true /p:CoverletOutput="../coverage.json" /p:MergeWith="../coverage.json" /maxcpucount:1 /p:CoverletOutputFormat="opencover" I received:

λ dotnet test /p:CollectCoverage=true /p:CoverletOutput="../coverage.json" /p:MergeWith="../coverage.json" /maxcpucount:1 /p:CoverletOutputFormat="opencover"
Test run for C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Api.IntegrationTests\bin\Debug\netcoreapp3.1\HappyCode.NetCoreBoilerplate.Api.IntegrationTests.dll(.NETCoreApp,Version=v3.1)
Microsoft (R) Test Execution Command Line Tool Version 16.3.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.

Test Run Successful.
Total tests: 5
     Passed: 5
 Total time: 4,8472 Seconds
Test run for C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Api.UnitTests\bin\Debug\netcoreapp3.1\HappyCode.NetCoreBoilerplate.Api.UnitTests.dll(.NETCoreApp,Version=v3.1)
Microsoft (R) Test Execution Command Line Tool Version 16.3.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.

Test Run Successful.
Total tests: 8
     Passed: 8
 Total time: 2,5702 Seconds

Calculating coverage result...
  Generating report '..\coverage.json'

+-----------------------------------+--------+--------+--------+
| Module                            | Line   | Branch | Method |
+-----------------------------------+--------+--------+--------+
| HappyCode.NetCoreBoilerplate.Api  | 13,19% | 11,86% | 19,23% |
+-----------------------------------+--------+--------+--------+
| HappyCode.NetCoreBoilerplate.Core | 0,93%  | 0%     | 3,17%  |
+-----------------------------------+--------+--------+--------+

+---------+-------+--------+--------+
|         | Line  | Branch | Method |
+---------+-------+--------+--------+
| Total   | 6,81% | 7,52%  | 7,86%  |
+---------+-------+--------+--------+
| Average | 7,06% | 5,93%  | 11,2%  |
+---------+-------+--------+--------+

Test run for C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\bin\Debug\netcoreapp3.1\HappyCode.NetCoreBoilerplate.Core.UnitTests.dll(.NETCoreApp,Version=v3.1)
Microsoft (R) Test Execution Command Line Tool Version 16.3.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.

Test Run Successful.
Total tests: 4
     Passed: 4
 Total time: 2,3510 Seconds

Calculating coverage result...
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error : Unexpected character encountered while parsing value: <. Path '', line 0, position 0. [C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.JsonTextReader.ParseValue() [C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.JsonTextReader.Read() [C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter) [C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent) [C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType) [C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings) [C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings) [C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Coverlet.Core.Coverage.GetCoverageResult() in C:\Users\toni\Workspace\coverlet\src\coverlet.core\Coverage.cs:line 224 [C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Coverlet.MSbuild.Tasks.CoverageResultTask.Execute() in C:\Users\toni\Workspace\coverlet\src\coverlet.msbuild.tasks\CoverageResultTask.cs:line 91 [C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]

So it counts coverage for the first project and saves it into json, but for the second project, it can not proceed.

Is it possible to count coverage and merge into one json (according to MergeWith parameter) and then convert the json results into opencover?

Dec 31 '19 10:12 lkurzyniec

@lkurzyniec can you try to run same command but with /p:CoverletOutputFormat="opencover,json" parameter?Similar to https://github.com/tonerdo/coverlet/issues/262#issuecomment-569572522 and remember to use /maxcpucount:1

Is it possible to count coverage and merge into one json (according to MergeWith parameter) and then convert the json results into opencover?

Yes I did same sample on our "sample" section https://github.com/tonerdo/coverlet/blob/master/Documentation/Examples/MSBuild/MergeWith/HowTo.md , you can clone repo and test solution, but you cannot run command "solution wide".

Jan 02 '20 21:01 MarcoRossignoli

For reference link msbuild task sync discussion https://github.com/microsoft/msbuild/issues/4898 https://github.com/microsoft/msbuild/issues/4954

Jan 02 '20 21:01 MarcoRossignoli

Unfortunately, it doesn't work

λ dotnet test /p:CollectCoverage=true /p:CoverletOutput="../coverage.json" /p:MergeWith="../coverage.json" /maxcpucount:1 /p:CoverletOutputFormat=\"json,opencover\"
Test run for C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Api.IntegrationTests\bin\Debug\netcoreapp3.1\HappyCode.NetCoreBoilerplate.Api.IntegrationTests.dll(.NETCoreApp,Version=v3.1)
Microsoft (R) Test Execution Command Line Tool Version 16.3.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.

Test Run Successful.
Total tests: 5
     Passed: 5
 Total time: 5,0513 Seconds
Test run for C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Api.UnitTests\bin\Debug\netcoreapp3.1\HappyCode.NetCoreBoilerplate.Api.UnitTests.dll(.NETCoreApp,Version=v3.1)
Microsoft (R) Test Execution Command Line Tool Version 16.3.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.

Test Run Successful.
Total tests: 8
     Passed: 8
 Total time: 3,1337 Seconds

Calculating coverage result...
  Generating report '..\coverage.json'
  Generating report '..\coverage.json'

+-----------------------------------+--------+--------+--------+
| Module                            | Line   | Branch | Method |
+-----------------------------------+--------+--------+--------+
| HappyCode.NetCoreBoilerplate.Api  | 13,19% | 11,86% | 19,23% |
+-----------------------------------+--------+--------+--------+
| HappyCode.NetCoreBoilerplate.Core | 35,51% | 35,71% | 44,44% |
+-----------------------------------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 24,81% | 19,54% | 37,07% |
+---------+--------+--------+--------+
| Average | 24,35% | 23,78% | 31,83% |
+---------+--------+--------+--------+

Test run for C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\bin\Debug\netcoreapp3.1\HappyCode.NetCoreBoilerplate.Core.UnitTests.dll(.NETCoreApp,Version=v3.1)
Microsoft (R) Test Execution Command Line Tool Version 16.3.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.

Test Run Successful.
Total tests: 4
     Passed: 4
 Total time: 2,9084 Seconds

Calculating coverage result...
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error : Unexpected character encountered while parsing value: <. Path '', line 0, position 0. [C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.JsonTextReader.ParseValue() [C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.JsonTextReader.Read() [C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter) [C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent) [C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType) [C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings) [C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings) [C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Coverlet.Core.Coverage.GetCoverageResult() in C:\Users\toni\Workspace\coverlet\src\coverlet.core\Coverage.cs:line 224 [C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Coverlet.MSbuild.Tasks.CoverageResultTask.Execute() in C:\Users\toni\Workspace\coverlet\src\coverlet.msbuild.tasks\CoverageResultTask.cs:line 91 [C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]

What's interesting here is that it generate json two times:

Calculating coverage result...
  Generating report '..\coverage.json'
  Generating report '..\coverage.json'

Jan 03 '20 13:01 lkurzyniec

Try with

dotnet test /p:CollectCoverage=true /p:CoverletOutput="../" /p:MergeWith="../coverage.json" /maxcpucount:1 /p:CoverletOutputFormat=\"json,opencover\"

Similar to https://stackoverflow.com/questions/53255065/dotnet-unit-test-with-coverlet-how-to-get-coverage-for-entire-solution-and-not/56281108#56281108 but with -m:1

For instance you can try with our samples on repo

D:\git\coverlet\Documentation\Examples\MSBuild\MergeWith
λ dotnet test /p:CollectCoverage=true  /p:CoverletOutput=../CoverageResults/ /p:MergeWith="../CoverageResults/coverage.json" /p:CoverletOutputFormat=\"opencover,json\" -m:1

Jan 03 '20 13:01 MarcoRossignoli

@MarcoRossignoli That works fine! In my case, it gives me solution wide results.

Summary

cmd command

dotnet test /p:CollectCoverage=true /p:CoverletOutput="../" /p:MergeWith="../coverage.json" /maxcpucount:1 /p:CoverletOutputFormat=\"json,opencover\"

gives me the same results as following two commands

dotnet test ./test/HappyCode.NetCoreBoilerplate.Core.UnitTests/HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj /p:CollectCoverage=true /p:CoverletOutput=../CoverageResults/

dotnet test ./test/HappyCode.NetCoreBoilerplate.Api.UnitTests/HappyCode.NetCoreBoilerplate.Api.UnitTests.csproj /p:CollectCoverage=true /p:CoverletOutput=../CoverageResults/ /p:MergeWith="../CoverageResults/coverage.json" /p:CoverletOutputFormat="opencover"

What's more, your command dotnet test /p:CollectCoverage=true /p:CoverletOutput=../CoverageResults/ /p:MergeWith="../CoverageResults/coverage.json" /p:CoverletOutputFormat=\"opencover,json\" -m:1 from https://github.com/tonerdo/coverlet/blob/master/Documentation/Examples/MSBuild/MergeWith/HowTo.md also works :)

Jan 08 '20 19:01 lkurzyniec

Glad to hear!

Jan 08 '20 20:01 MarcoRossignoli

Not sure if this should be a separate question, or asked here, but is it possible to do solution wide coverage using the coverlet .Net global tool? I'm using coverlet in Azure Pipelines and would like it to be system agnostic (Linux, Windows, Mac). The task I'm running looks like this:

  - script: coverlet $(buildArtifact)/tests/core/My.Project.Tests/bin/$(buildConfiguration)/netcoreapp3.1/My.Project.Tests.dll --target "dotnet" --targetargs "test $(buildArtifact)/tests/core/My.Project.Tests --no-build" --format cobertura
    displayName: Run Coverlet to get code coverage

But could it target a directory and run all tests projects under that directory? Or target a solution file?

Jan 30 '20 09:01 robertlarkins

I'm using coverlet in Azure Pipelines and would like it to be system agnostic (Linux, Windows, Mac)

Why don't use collectors https://github.com/tonerdo/coverlet/blob/master/Documentation/VSTestIntegration.md? Why .NET tool?

Coverlet is cross platform and works cross plat for every "drivers" you're using https://github.com/tonerdo/coverlet#quick-start but the collectors is the preferred way to consume due to integration with vstest platform and you can run command on sln(it generates one coverage file for every test project you have in solution).

Jan 30 '20 10:01 MarcoRossignoli

At this stage I was only using coverlet as part of the Azure Pipeline, so didn't think the test projects should know or depend on it. Will try installing coverlet.collector as a NuGet package.

Jan 30 '20 23:01 robertlarkins

So I added coverlet.collector as a NuGet package and got it going for a single test project with this:

  - task: DotNetCoreCLI@2
    displayName: 'Run tests for $(solution) with Cobertura'
    inputs:
      command: test
      projects: '$(buildArtifact)/tests/**/*.csproj'
      arguments: -c $(buildConfiguration) --collect:"XPlat Code Coverage"

  - task: PublishCodeCoverageResults@1
    displayName: 'Publish code coverage'
    inputs:
      codeCoverageTool: cobertura
      summaryFileLocation: '$(Agent.TempDirectory)/*/coverage.cobertura.xml'

But this produces a separate coverage.cobertura.xml file per test project (each in there own guid directory). In this Azure Pipelines context how would I get the individual test results merged together so that a single code coverage publish can be done?

Jan 31 '20 04:01 robertlarkins

@robertlarkins my suggestion would be to read both this article and whole this thread. Then try to achieve solution-wide results locally. After that move your solution to Azure DevOps.

Jan 31 '20 08:01 lkurzyniec

@lkurzyniec Will attempt and see how I get on.

Jan 31 '20 08:01 robertlarkins

At the moment I've removed from documentation Merge support for collectors because we're working on better approach https://github.com/tonerdo/coverlet/pull/704 (merge works well only for sequential tests), because "as is" doesn't work(vstest generates different filenames as you can see).

Actually that parameters is "hidden" but usable, so if you want be sure to not hit https://github.com/tonerdo/coverlet/blob/master/Documentation/KnowIssues.md#1-vstest-stops-process-execution-earlydotnet-test and use merge with vstest you can use a trick of our contributor @p4p3 that with a script move generated coverage files for every test run https://github.com/tonerdo/coverlet/pull/225#issuecomment-573896446

If you don't hit issue above you can use msbuild driver and https://github.com/tonerdo/coverlet/blob/master/Documentation/Examples/MSBuild/MergeWith/HowTo.md

Jan 31 '20 12:01 MarcoRossignoli

@lkurzyniec , @MarcoRossignoli Hi. I am trying to execute the above command but getting an error

PS > dotnet test /p:CollectCoverage=true /p:CoverletOutput=../.coveragedata/cover
age /p:MergeWith="../coverage.json" /p:CoverletOutputFormat=\"json,opencover\" -m:1
MSBUILD : error MSB1006: Property is not valid.
Key: opencover\

Apr 16 '20 17:04 sungam3r

@sungam3r you need to translate characters on powershell take a look https://github.com/tonerdo/coverlet/blob/master/Documentation/MSBuildIntegration.md#note-for-powershell--vsts-users %2c for ,

Apr 16 '20 17:04 MarcoRossignoli

Wow! Thanks. I read this section of the documentation, but for some reason I missed it.

Apr 16 '20 17:04 sungam3r

No prob

Apr 16 '20 17:04 MarcoRossignoli

Still having issues in getting merge results and allowing to run ReportGenerator from that. Following is the dotnet test command I am trying to run. Using coverlet 2.9.0 in my test projects. Getting the merge results works fine untill I add that last part regarding output format. But if I only output the default json I cannot get the ReportGenerator to produce anything valid. And if I use other formats in dotnet test, I cannot seem to be able to get results merged.

rm -Recurse CoverageResults/* ; dotnet test /p:CollectCoverage=true /p:Exclude="*UnitTests" /p:MergeWith="../../CoverageResults/coverage.json" /maxcpucount:1 /p:CoverletOutput='../../CoverageResults/coverage.json' /p:CoverletOutputFormat=\"json%2copencover\"

C:\Users\username\.nuget\packages\coverlet.msbuild\2.9.0\build\coverlet.msbuild.targets(60,5): error : Unexpected character encountered while parsing value: <. Path '', line 0, position 0. [C:\MyProject\Source\MyCompany.Domain.UnitTests\MyCompany.Domain.UnitTests.csproj] C:\Users\username\.nuget\packages\coverlet.msbuild\2.9.0\build\coverlet.msbuild.targets(60,5): error : at Newtonsoft.Json.JsonTextReader.ParseValue() [C:\MyProject\Source\MyCompany.Domain.UnitTests\MyCompany.Domain.UnitTests.csproj] C:\Users\username\.nuget\packages\coverlet.msbuild\2.9.0\build\coverlet.msbuild.targets(60,5): error : at Newtonsoft.Json.JsonReader.ReadAndMoveToContent() [C:\MyProject\Source\MyCompany.Domain.UnitTests\MyCompany.Domain.UnitTests.csproj] C:\Users\username\.nuget\packages\coverlet.msbuild\2.9.0\build\coverlet.msbuild.targets(60,5): error : at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ReadForType(JsonReader reader, JsonContract contract, Boolean hasConverter) [C:\MyProject\Source\MyCompany.Domain.UnitTests\MyCompany.Domain.UnitTests.csproj] C:\Users\username\.nuget\packages\coverlet.msbuild\2.9.0\build\coverlet.msbuild.targets(60,5): error : at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent) [C:\MyProject\Source\MyCompany.Domain.UnitTests\MyCompany.Domain.UnitTests.csproj] C:\Users\username\.nuget\packages\coverlet.msbuild\2.9.0\build\coverlet.msbuild.targets(60,5): error : at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType) [C:\MyProject\Source\MyCompany.Domain.UnitTests\MyCompany.Domain.UnitTests.csproj]

Oct 21 '20 11:10 aviita

And I have tried different variations of this CoverletOutputFormat string, all resulting to error of some sort with command above:

/p:CoverletOutputFormat=\"json%2copencover\" 

/p:CoverletOutputFormat="json%2copencover"

/p:CoverletOutputFormat='json,opencover'

/p:CoverletOutputFormat='json%2copencover'

Oct 21 '20 11:10 aviita

And found my problem, so I am not supposed to put the json file, but only a path to CoverletOutput param. Defining json file likely caused coverlet xml to be written to the json file:

rm -Recurse CoverageResults/* ; dotnet test /p:CollectCoverage=true /p:Exclude="*UnitTests" /p:MergeWith="../../CoverageResults/coverage.json" /maxcpucount:1 /p:CoverletOutput='../../CoverageResults/' /p:CoverletOutputFormat='\"json%2copencover\"'

Oct 21 '20 11:10 aviita

@aviita sorry for the delay(very busy time), glad to hear you solve your problem.

Nov 10 '20 08:11 MarcoRossignoli

https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/Examples/MSBuild/MergeWith/HowTo.md

It was helpfull for me. I use coverall and special solutions for unit and functional tests. https://github.com/kreghek/Zilon_Roguelike/blob/362997fa7216e1ba77c18ea120580e29b4ada16e/.github/workflows/coverall.yml

Jan 12 '21 16:01 kreghek

Solution-wide coverage reports seem to work well using MergeWith, but Threshold doesn't seem to work the way I'd expect with multiple projects. For example, if I do something like this:

dotnet test /p:CollectCoverage=true /p:CoverletOutput="../CoverageResults/" /p:MergeWith="../CoverageResults/coverage.json" /p:CoverletOutputFormat="\"cobertura,json\"" /p:Threshold=$MIN_CODE_COVERAGE_PERCENTAGE /p:ThresholdType=line /p:ThresholdStat=total

The solution-wide reports seem to be okay, but the threshold is checked after each test project is run, on the total coverage so far. Depending on the order test projects are run in this can cause our CI pipeline to fail even if overall coverage is actually above the threshold.

I found a workaround that involves running tests multiple times, but that's kind of hacky and adds unnecessary run time. Is there a better supported approach to accurate solution-wide thresholds? #598 seems relevant, but it was closed as a duplicate of this one so I'm asking here. Thanks in advance for any advice, and thanks for your work maintaining coverlet.

Sep 24 '21 23:09 mgiles

coverlet coverlet copied to clipboard

Solution wide coverage

Summary

coverlet
coverlet copied to clipboard