`SetXPlatDataCollectorPath` target is executed stochastically, thus coverage sometimes is not generated

[andrei-epure-sonarsource]

TL;DR

When adding coverlet into our CI pipe (to replace OpenCover), it has stochastic behavior in generating code coverage files.

The setup

PR with the changes to our CI to integrate coverlet: https://github.com/SonarSource/sonar-dotnet/pull/5913

Versions:

• OS: Windows Server 2019 Datacenter (via Google Cloud Platform)
• dotnet 6.0.203
• MSBuild 17.2.1.25201
• "Microsoft.NET.Test.Sdk" Version="16.11.0"
• "coverlet.collector" Version="3.1.2"
• "coverlet.MSBuild" Version="3.1.2" (note: I've added this only as a last resort during my experiments; however, with or without it, it's the same behavior)

CI setup: we have an Azure Pipeline where we:

• first, do a build (job: dotnetBuildjob). We then upload the binaries
• then start in multiple parallel jobs:
  • run net48 UTs (job: runUnitTestsJobNet48)
  • run net6.0 UTs (job: runUnitTestsJobNet6)
  • for both, we upload code coverage reports (XML files in opencover format) and test execution reports (.trx files)
• after the two UT jobs are done, we download the tests and coverage reports, and we run SonarCloud analysis (job: dotNetAnalysis)
• note: the virtual machines we use are reused and are not created from scratch
• note: as the two jobs are running independently and in parallel, they should not affect each other

Other setup details:

• a single UT project with :
  • <TargetFrameworks>net6.0;net48</TargetFrameworks>
  • <IsTestProject>true</IsTestProject> to overcome vstest#3790 - we’ve had similar behavior like the one described in this issue, but for the IsTestProject property not being set (and because of that, UTs wouldn't execute at all). Putting IsTestProject in our UT project csproj is the workaround.
• a runsettings.xml with <Format>opencover</Format> and InProcDataCollector as specified in your Known Issues doc
• we invoke coverlet: dotnet test .\analyzers\SonarAnalyzer.sln --no-build --collect:"XPlat Code Coverage" --settings .\analyzers\runsettings.xml -l "trx;logfilename=net48.trx" -l "console;verbosity=diagnostic" -c $(BuildConfiguration) -f net48 -v diag

The problem

Observed behavior:

• The tests are invoked correctly for both jobs (no stochastic behavior here). The trx files get generated all the time.
• only one of the net48 or the net6.0 job generates code coverage (it's always 1/2 jobs generating), and it's stochastic: sometimes the net48 job generates ccov, sometimes net6.0 does (and the other does not). The code coverage XML files get generated stochastically.
• if we restart the failing job, it behaves the same way

Why the coverage does not get created

When comparing the diagnostic logs of a net6.0 job with no coverage vs. a net6.0 job with coverage, we've noticed the SetXPlatDataCollectorPath target does not get executed, so the coverage does not get created.

More specifically, when coverage is present:

2022-07-26T09:24:17.6603175Z Target Performance Summary:
2022-07-26T09:24:17.6603834Z         0 ms  ValidateProjects                           1 calls
2022-07-26T09:24:17.6604558Z         0 ms  ValidateToolsVersions                      1 calls
2022-07-26T09:24:17.6605511Z         0 ms  SetXPlatDataCollectorPath                  1 calls
2022-07-26T09:24:17.6606227Z        12 ms  GenerateProgramFile                        1 calls
2022-07-26T09:24:17.6606976Z        13 ms  ValidateSolutionConfiguration              1 calls
2022-07-26T09:24:17.6607708Z        33 ms  ShowCallOfVSTestTaskWithParameter         13 calls
2022-07-26T09:24:17.6608526Z       174 ms  ShowInfoMessageIfProjectHasNoIsTestProjectProperty  12 calls
2022-07-26T09:24:17.6609236Z     75223 ms  VSTest                                    15 calls

When coverage is not present:

2022-07-26T09:05:20.0729428Z Target Performance Summary:
2022-07-26T09:05:20.0730132Z         0 ms  ValidateProjects                           1 calls
2022-07-26T09:05:20.0730805Z         0 ms  ValidateToolsVersions                      1 calls
2022-07-26T09:05:20.0731456Z        17 ms  GenerateProgramFile                        1 calls
2022-07-26T09:05:20.0732127Z        38 ms  ShowCallOfVSTestTaskWithParameter         13 calls
2022-07-26T09:05:20.0732889Z       151 ms  ShowInfoMessageIfProjectHasNoIsTestProjectProperty  12 calls
2022-07-26T09:05:20.0733624Z       225 ms  ValidateSolutionConfiguration              1 calls
2022-07-26T09:05:20.0734256Z     49154 ms  VSTest                                    15 calls

SetXPlatDataCollectorPath is the missing link.

Please see the attached logs:

• 2022-07-26-net6_trx_with_ccov_REDACTED.txt - diagnostic logs when the code coverage gets generated for the net6.0 job
• 2022-07-26-net6_trx_no_ccov_REDACTED.txt - diagnostic logs when the code coverage does not get generated for the net6.0 job

Necessary help

We need guidance on how to proceed further with investigating this. The SetXPlatDataCollectorPath target does not have any preconditions; it should just get executed before VSTest. I have no idea how to debug why this happens - I see no hints in the diagnostic logs.

I realize the issue might be related to whatever the root cause behind vstest#3790, but I do not know how to pinpoint the root cause.

Thank you for supporting this tool and for providing us with the necessary help on how to debug this further.

[MarcoRossignoli]

To debug you should run the same command with /bl argument and analyze imports using https://msbuildlog.com/

[andrei-epure-sonarsource]

@MarcoRossignoli is there a difference between the information contained in the diagnostic logs and the binary logs?

[MarcoRossignoli]

Yep, binary logs contains msbuild information so there you can understand why target(msbuild concept) doesn't run...test logs are emitted by code at runtime but the problem of targets is solved by msbuild not by test platform

[andrei-epure-sonarsource]

For reference, in case someone else had this problem, we ended up using AltCover, which worked for us (PR#6171).

I'm going to keep this issue in my backlog to go back to this investigation in order to help the coverlet team identify the root cause. I'll make some progress when I get some spare time.

[MarcoRossignoli]

FYI dotnet test at the moment supports by design dynamic code coverage running --collect "Code Coverage" and it's supported in Windows (x86, x64 and Arm64), Linux (x64) and macOS (x64)

https://learn.microsoft.com/en-us/dotnet/core/tools/dotnet-test

there's also a .NET Tool for E2E testing https://learn.microsoft.com/en-us/dotnet/core/additional-tools/dotnet-coverage