.NET Code Coverage no longer being generated - 20221119.2

[HatakuSec]

Description

We are noticing the same issue as identified in https://github.com/actions/runner-images/issues/6588 with the latest current image. All our builds with "20221119.2" have been failing to produce any code coverage during their test run tasks,

As specified in the above ticket, our code coverage tasks is as follows:

- task: DotNetCoreCLI@2
  displayName: Run unit and service tests
  inputs:
     command: test
     projects: |
       app/**/*ServiceTests/*.csproj
       app/**/*UnitTests/*.csproj
     arguments: '--configuration ${{ parameters.buildConfiguration }} /p:CollectCoverage=true /p:ThresholdStat=total /p:Exclude=[xunit.*]* /p:CoverletOutputFormat=json%2ccobertura /p:CoverletOutput=$(Build.SourcesDirectory)/TestResults/Coverage/ /p:MergeWith=$(Build.SourcesDirectory)/TestResults/Coverage/coverage.json /maxcpucount:1 "--logger:\"console;verbosity=detailed\""'
     nobuild: true
     publishTestResults: true
     testRunTitle: Run unit and service test

We ran our tests locally with and without .NET 7.0.100 it seems like when the 7.0 SDK is used with Microsoft (R) Test Execution Command Line Tool Version 17.4.0 (x64) no code coverage report is created however when 7.0.100 is uninstalled, 6.0 SDK is used and the Command Line Tool Version 17.3.1 is used we get code coverage reports created.

Platforms affected

• [X] Azure DevOps
• [ ] GitHub Actions - Standard Runners
• [ ] GitHub Actions - Larger Runners

Runner images affected

• [ ] Ubuntu 18.04
• [X] Ubuntu 20.04
• [ ] Ubuntu 22.04
• [ ] macOS 10.15
• [ ] macOS 11
• [ ] macOS 12
• [ ] Windows Server 2019
• [ ] Windows Server 2022

Image version and build link

20221119.2

Is it regression?

no

Expected behavior

Code coverage calculated and report generated after test runs

Actual behavior

Test run without any issue however the code coverage doesn't get generated causing failures on further steps of our pipelines

Repro steps

Run dotnet test via the DotNetCLI@2 task using image version 20221119.2

[igorboskovic3]

Hi @HatakuSec , thank you and could you provide minimal repro steps?

[bjornbouetsmith]

Hi,

This is caused by this issue in dotnet: https://github.com/microsoft/vstest/issues/4014

Which strips away all /p:Property=value.

So can you please roll back your image change for .NET SDK 7 and remove it until this bug has been fixed, since it renders all codecoverage more or less impossible.

P.S. It also affects all Ubuntu images that includes .NET SDK 7.0.100.

It seems like Microsoft did not fix the bug and still released SDK 7.0.100, even though the bug was reported before release date.

So it would be great if you could either roll back your .NET SDK change, so SDK 7 gets removed from all images - or roll forward to a version without .NET SDK 7 in it.

My project also fails: https://github.com/bjornbouetsmith/ROOT.Zfs/actions/runs/3543814503/jobs/5950580399

Alternatively then you enhance images, so we can specify a specific version, so we can get last good version, i.e. 20221027 - which does not have .NET SDK 7 installed.

Thank you

[bjornbouetsmith]

A work around until you either remove SDK 7 from the image or updates it with a working SDK 7 is to include a global.json in the root of your git repository:

{
  "sdk": {
    "version": "6.0.100",
    "rollForward": "latestMinor"
  }
}

That should force the dotnet runtime to stay at 6.0 even if a higher version is installed - and it works for me - but I would prefer if the image was rolled back or forward without SDK 7

[RetainPhil]

Same thing for us - since it looks like our VM images were all upgraded from Current image version: '20221027.1' to Current image version: '20221120.1' yesterday which now has VS 17.4 and .net7 sdk installed and hence, all our build pipelines are now failing due to coverage output not generating from dotnet test. I've had to disable our coverage checks for now...

[mikhailkoliada]

DotNetCoreCLI@2 does not pick up a .NET version for you, it uses the latest available which is .NET 7 right now installed with the updated Visual Studio. Nothing prevents from switching to any older .NET version as far as I can judge. To do that, please use the UseDotNet@2 task

[bjornbouetsmith]

To do that, please use the UseDotNet@2 task

That might be possible, but it is unfortunate that the runner images have been upgraded with a broken .NET SDK version. I get that it should be upgraded as soon as a new .NET SDK is out, but this time it would be great to hold back, since the default behavior in calling dotnet is that it uses the latest SDK and that is broken. And then everyone has to make tweaks to actions or their code to work around this behavior until the runner image is upgraded again with a working SDK.

It would be much easier for everyone affected by this, if you just rolled back the change - and removed the .NET SDK 7 from the image until the bug has been fixed by microsoft.

Or alternatively - let people chose what version of the runner-image to use - so people always could "lock" to a last known version.

[bjornbouetsmith]

DotNetCoreCLI@2 does not pick up a .NET version for you, it uses the latest available which is .NET 7 right now installed with the updated Visual Studio. Nothing prevents from switching to any older .NET version as far as I can judge. To do that, please use the UseDotNet@2 task

Also is this not for azure pipelines - we are discussing github actions here?

[mikhailkoliada]

@bjornbouetsmith unfortunately we can not downgrade .NET version at all, though we are in touch with the .NET team to see fix rolled as quickly as possible. For the time being, I think workaround you suggested here is viable.

[bjornbouetsmith]

@bjornbouetsmith unfortunately we can not downgrade .NET version at all, though we are in touch with the .NET team to see fix rolled as quickly as possible. For the time being, I think workaround you suggested here is viable.

What do you mean you cannot downgrade .NET? That does not make sense at all - you(github) have released a runner-image that is somewhat broken - the correct thing to do is roll back - and if people urgently want .NET 7 - they can just call the script that downloads and install SDK 7, then the rest of us can continue with non-broken builds.

Also all these issues could have been avoided if you made it possible to select a specific version of runner-images instead of force upgrading everyone to latest version.

[Coder3333]

I have found that /p: has not worked for a while, not even in .Net 6. The "--" and "-" syntax is respected.

--collect:"Code Coverage" --settings:"runsettings.runsettings" -c:$(BuildConfiguration)

[xPaw]

.NET SDK 7.0.101 was released, can it be updated in the runner image, which should fix this issue?

[Evangelinexx]

To be honest I would have preferred not installing a specific version of dotnet on the runner images at all so that we could all manually install just the version of dotnet that we need. dotnet 7 isn't even an LTS version.

[fgheysels]

I'm currently no longer using coverlet.msbuild, but I'm using coverlet.collector and I'm using the arguments as mentionned by @Coder3333 .

- task: DotNetCoreCLI@2
  displayName: 'Run testsuite'
  inputs:
    command: 'test'
    projects: |
      src\SomeProject\someproject.csproj      
    testRunTitle: 'someproject tests'
    arguments: --collect:"XPlat Code Coverage" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=cobertura
    publishTestResults: true

- task: reportgenerator@5
  displayName: Generate code coverage report for SomeProject
  inputs:
    title: SomeProject coverage report
    reports: $(Agent.TempDirectory)/*/coverage.cobertura.xml
    targetdir: $(Build.SourcesDirectory)/TestCoverageReport/

- task: PublishCodeCoverageResults@1
  displayName: 'Publish code coverage report'
  inputs:
    codeCoverageTool: 'Cobertura'
    summaryFileLocation: '$(Build.SourcesDirectory)/TestCoverageReport/*.xml'

[mikhailkoliada]

The fix has been released and images updated, too.

[devdorn]

Hi @mikhailkoliada ,

just tested the fix and the issue remains. Image version: '20221212.1'.

I'm using ubuntu-latest.

My Test-Task (I'm using 2 Formats, because I also push to Sonar Cloud):

- task: DotNetCoreCLI@2
    displayName: "dotnet test"
    inputs:
        command: "test"
        testRunTitle: "Non integration tests"
        arguments: "--configuration $(buildConfiguration) --filter Category!=Integration /p:CollectCoverage=true /p:CoverletOutputFormat=opencover%2cCobertura /p:CoverletOutput=$(Build.SourcesDirectory)/Coverage/"       
        publishTestResults: true
        projects: "$(test)"

My Code-Coverage Task:

- task: PublishCodeCoverageResults@1
    displayName: "Publish code coverage report"
    inputs:
        codeCoverageTool: "Cobertura"
        summaryFileLocation: "$(Build.SourcesDirectory)/**/coverage.cobertura.xml"

The Code Coverage Results are only generated when I use the UseDotNet task with .NET 6 SDK at the start of my pipeline.

[mikhailkoliada]

Hmm, right, I missed the fact that ubuntu has not (yet) got its update. It will be done with the next image release, which is next week

[mikhailkoliada]

Should be finally fixed.

[Milujem]

Malo by byť konečne opravené.