Feature Request: Allow threshold minimums to be applied only to the current module when mergeWith is used

[rcollette]

I use MergeWith together with Threshold. I want to specify that the threshold applies to the module currently being tested , such that modules that appear in the coverage table only because they were merged, are not subject to the threshold requirements of the current test run.

The reason I want to do this is because we expect the thresholds on the Repository to actually go down over time as more code is added. We only unit test some helper functionality within the repository and not all the queries (as per Microsoft advice on unit testing repositories and EF).

Without being able to be specific as to what module the thresholds apply to, ideally the current one, I wind up having to adjust the thresholds on every other test run that depends on the Repository project, whenever code is added to the Repository. It is a frequent and major pain point.

This also forces me to set thresholds on totals rather than minimums because the minimums apply to all modules and not just the module currently being tested. I would much rather set a minimum that applies only to the current module being tested, when using MergeWith.

Perhaps a flag to enable this behavior would perform the merge operation after all other logic has run, would be an easy path to making this work.

We generate tests using commands like

dotnet test "Api.HttpClient.Test/Api.HttpClient.Test.csproj" -p:CollectCoverage=true -p:CoverletOutput='../TestResults/' -p:CoverletOutputFormat=\"cobertura%2cjson\" -p:Threshold=\"33%2c3.84%2c42.5\" -p:ThresholdType=\"line%2cbranch%2cmethod\" -p:ThresholdStat=total -p:ThresholdStat=total -p:DeterministicSourcePaths=true -m:1
dotnet test "Api.Repository.Test/Api.Repository.Test.csproj" -p:CollectCoverage=true -p:CoverletOutput='../TestResults/' -p:MergeWith="../TestResults/coverage.json" -p:CoverletOutputFormat=\"cobertura%2cjson\" -p:Threshold=\"6%2c22%2c37\" -p:ThresholdType=\"line%2cbranch%2cmethod\" -p:ThresholdStat=total -p:ThresholdStat=total -p:DeterministicSourcePaths=true -m:1
dotnet test "Api.Service.Test/Api.Service.Test.csproj" -p:CollectCoverage=true -p:CoverletOutput='../TestResults/' -p:MergeWith="../TestResults/coverage.json" -p:CoverletOutputFormat=\"cobertura%2cjson\" -p:Threshold=\"13%2c57%2c63\" -p:ThresholdType=\"line%2cbranch%2cmethod\" -p:ThresholdStat=total -p:ThresholdStat=total -p:DeterministicSourcePaths=true -m:1
dotnet test "Api.Web.Test/Api.Web.Test.csproj" -p:CollectCoverage=true -p:CoverletOutput='../TestResults/' -p:MergeWith="../TestResults/coverage.json" -p:CoverletOutputFormat=\"cobertura%2cjson\" -p:Threshold=\"15%2c52%2c63\" -p:ThresholdType=\"line%2cbranch%2cmethod\" -p:ThresholdStat=total -p:ThresholdStat=total -p:DeterministicSourcePaths=true -m:1

This results in coverage tables like

HTTP Client

+------------------------------+--------+--------+--------+
| Module                       | Line   | Branch | Method |
+------------------------------+--------+--------+--------+
| Api.HttpClient | 45.67% | 23.68% | 42.28% |
+------------------------------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 45.67% | 23.68% | 42.28% |
+---------+--------+--------+--------+
| Average | 45.67% | 23.68% | 42.28% |
+---------+--------+--------+--------+

Merged With repository

+------------------------------+--------+--------+--------+
| Module                       | Line   | Branch | Method |
+------------------------------+--------+--------+--------+
| Api.Repository | 10.45% | 21.42% | 33.12% |
+------------------------------+--------+--------+--------+
| Api.HttpClient | 45.67% | 23.68% | 42.28% |
+------------------------------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 12.5%  | 21.81% | 36.01% |
+---------+--------+--------+--------+
| Average | 28.06% | 22.55% | 37.7%  |
+---------+--------+--------+--------+

Merged with Service

+------------------------------+--------+--------+--------+
| Module                       | Line   | Branch | Method |
+------------------------------+--------+--------+--------+
| Api.Repository | 11.41% | 35.16% | 45.82% |
+------------------------------+--------+--------+--------+
| Api.HttpClient | 48.31% | 23.68% | 49.66% |
+------------------------------+--------+--------+--------+
| Api.Service    | 89.73% | 78.87% | 89.54% |
+------------------------------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 27.75% | 58.92% | 65.25% |
+---------+--------+--------+--------+
| Average | 49.81% | 45.9%  | 61.67% |
+---------+--------+--------+--------+

Finally, merged with web

+------------------------------+--------+--------+--------+
| Module                       | Line   | Branch | Method |
+------------------------------+--------+--------+--------+
| Api.Repository | 11.41% | 35.16% | 45.82% |
+------------------------------+--------+--------+--------+
| Api.Web        | 47.75% | 35.52% | 58.77% |
+------------------------------+--------+--------+--------+
| Api.HttpClient | 48.31% | 23.68% | 49.66% |
+------------------------------+--------+--------+--------+
| Api.Service    | 89.97% | 79.22% | 90.11% |
+------------------------------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 29.38% | 53.65% | 64.68% |
+---------+--------+--------+--------+
| Average | 49.36% | 43.39% | 61.09% |
+---------+--------+--------+--------+

[MarcoRossignoli]

Thanks for the suggestion.

[github-actions[bot]]

This issue is stale because it has been open for 3 months with no activity.

[rcollette]

But still relevant

[github-actions[bot]]

This issue is stale because it has been open for 3 months with no activity.

[rcollette]

But still relevant

[github-actions[bot]]

This issue is stale because it has been open for 3 months with no activity.

[rcollette]

And still relevant