sonar-dotnet
sonar-dotnet copied to clipboard
SonarSource
Code Coverage: exclusions in .runsettings file are ignored
Description
after upgrading SonarQube Server from 5.6 to 6.7.4.38452 and C# plugin to 7.2 (build 5463), Code Coverage is still imported from VSTS build, but provided exclusions in .runsettings file for VSTest are ignored. With SonarQube 5.6 the .runsettings file exclusions were correctly applied.
Repro steps
VSTS SonarQube Build tasks used. VSTest task has Code Coverage enabled and a .runsettings file specified in the property "Settings file".
Expected behavior
Code Coverage in SonarQube should recognize exlusions from VSTest .runsettings file.
Actual behavior
Code Coverage is imported into SonarQube, but exclusions are not recognized.
Known workarounds
Related information
- SonarC# Version : 7.2
- Visual Studio Version : 15.7.3
- Scanner for MSBuild: VSTS Tasks Version 3
Has anyone looked into where this change would need to be made? I would like to see if I can get this fixed/added, but I can't seem to locate where.
Hi there,
I finally managed to spend some time looking at this issue.
Starting with the version 6.6 of SonarC#, we are now processing a new metric called "executable lines of code" which is used in addition to the coverage info to track lines covered/uncovered. As we don't process (and we never did) the .runsettings file, all files excluded in this configuration won't have any coverage info BUT using the executable lines of code will appear as un-covered on SonarQube.
As a short-term answer, you will need to provide a new parameter (sonar.coverage.exclusion=LIST_OF_FILES) to the scanner for msbuild (or the prepare task of TFS/VSTS) OR configure your project on the SonarQube/SonarCloud project administration page to exclude all files you have excluded in your .runsettings file. Note that you can only exclude a file or a group of files BUT not something with a lower granularity.
As a mid-term/long-term solution, we will investigate whether we should automatically exclude all files excluded in the .runsettings file.
Before moving forward with this, we'll have to check the impact of recently handling of .gitignore, and then come back with a plan
I'm coming back on this topic after a long pause (sorry for that!).
@berndku @rliberoff @millusdk @puddlewitt @JFMG @StingyJack For clarification of your usecase(s): are you looking for excluding projects / files just for code coverage or from analysis in general? In other words:
- would your expectation be for us to read the runsettings.xml file and exclude those files completely from analysis (like the behavior @Evangelink mentioned here and in the docs, but out of the box based on what's excluded in the runsettings file)
- or would you expect these files to be analyzed by our analyzers, but excluded only from code coverage?
Thank you 🙏
Would this apply to SonarCloud as well?
@StingyJack - if this were implemented, yes, it would apply for both SonarQube and SonarCloud.
Just excluded from code coverage analysis, which is what that runsettings file declares for the VS enterprise code coverage tooling.
I agree with the previous comments. The runsettings file should just do what it says in the official documentation https://docs.microsoft.com/en-US/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file That means just exclude everything what is described in the runsettings file from code coverage.
At the bottom of that docs page @JFMG shared is a link to this page that details the types that can be included or excluded for coverage analysis.
In the past I've needed three of those filters.
ModulePath - to either whitelist the assemblies to check or to exclude unit tests from being analyzed for coverage
Attribute - to filter out the ones mentioned in the example runsettings
Source - to exclude code gen files by extension (*.g.cs)
Thanks for the replies, @berndku , @JFMG , @StingyJack.
I'm thinking out loud: if we were to implement a partial solution (we like to deliver value incrementally) , for example starting with the Source filter only:
- would you find it more confusing that the Source filter is taken into consideration, and the ModulePath and Attribute (and others) are not?
- would you check the build logs in case the behavior from SonarQube/SonarCloud is different than what you'd expect (we'd mention in the build logs that only filter X is supported, and the rest are ignored)?
Another question: in my understanding, there would be only one .runsettings file per analysis, right?
Hi @mack0196 - this ticket is still open. I've asked a few questions here which I didn't get reply to.
We don't have this on our near-future roadmap.
(cc @tom-howlett-sonarsource )
@andrei-epure-sonarsource to answer your questions (didnt see them, sorry)
The Attribute filter and module path are usually the thing I would use the most, because historically I've had to exclude unit test projects themselves from having coverage reported on them (which is silly but true), and add attributes like for generated code (and exclude from code coverage). The Source is only when I need to exclude a generated file that doesnt use attributes to denote that its generated (like grpc.net used to but that should be fixed now).
I would expect there to be logs somewhere about what was included or not, at least when diagnostics or system.debug pipeline variable was enabled.
If we are doing it right, there is only one runsettings for all the projects in the solution (and repo). There shouldnt be a great need to test projects in the same repo differently that couldnt be handled by accommodating for it tin the runsettings, but there may be a case for it somewhere.
My team and I are still interested if you could have sonar respect the exclusions in our runsettings file.
@andrei-epure-sonarsource Any update on when this will be picked up by sonar? At the moment we need to add exclusion to both CodeCoverage.runsettings and in sonar cloud > General settings > B. Code Coverage Exclusions
This is the top voted issue on this tracker and is nearing it's 7th birthday. Is there any timeline on when or if this will be prioritised? The alternatives of setting sonar.coverage.exclusions is lacking in what can be configured with the limited regex matching patterns as you can't easily include a module path like what can be set in runsettings file. This could be mitigated with a property such as sonar.coverage.inclusions but this does not exist today.
