coverlet icon indicating copy to clipboard operation
coverlet copied to clipboard
verified publisher icon coverlet-coverage

[BUG] Crash during instrumentation

Open bart-vmware opened this issue 6 months ago • 9 comments

Describe the bug

Coverage is not being collected for one of our projects. With diagnostics turned on, I found the following exception, which is causing the problem:

TpTrace Warning: 0 : 38412, 1, 2025/06/30, 14:56:20.356, 3656098421787, datacollector.dll, [coverlet]Unable to instrument module: C:\Source\Projects\CoverletCrashDemo\TestProject1\bin\Release\net8.0\ClassLibrary1.dll
System.NullReferenceException: Object reference not set to an instance of an object.
   at Coverlet.Core.Instrumentation.Instrumenter.CollectLambdaMethodsInsideLocalFunction(MethodDefinition methodDefinition)+MoveNext() in /_/src/coverlet.core/Instrumentation/Instrumenter.cs:line 831
   at System.Collections.Generic.List`1.AddRange(IEnumerable`1 collection)
   at Coverlet.Core.Instrumentation.Instrumenter.InstrumentType(TypeDefinition type) in /_/src/coverlet.core/Instrumentation/Instrumenter.cs:line 500
   at Coverlet.Core.Instrumentation.Instrumenter.InstrumentModule() in /_/src/coverlet.core/Instrumentation/Instrumenter.cs:line 244
   at Coverlet.Core.Instrumentation.Instrumenter.Instrument() in /_/src/coverlet.core/Instrumentation/Instrumenter.cs:line 148
   at Coverlet.Core.Coverage.PrepareModules() in /_/src/coverlet.core/Coverage.cs:line 134

To Reproduce

  1. Extract the attached minimal repro project.
  2. Run: git clean -xdf && dotnet test --configuration Release --collect "XPlat Code Coverage" --logger trx --settings coverlet.runsettings -verbosity:diagnostic -noconsolelogger --diag:Logs/log.txt
  3. Search for NullReferenceException in the Logs directory and observe the stack trace from above.

CoverletCrashDemo.zip

Expected behavior Coverage is collected, no crash.

Actual behavior No coverage, warning in the log about failed instrumentation.

Configuration (please complete the following information): Please provide more information on your .NET configuration: * Which coverlet package and version was used? coverlet.collector 6.0.4 * Which version of .NET is the code running on? .NET 8 * What OS and version, and what distro if applicable? Windows 11 * What is the architecture (x64, x86, ARM, ARM64)? x64 * Do you know whether it is specific to that configuration? Also no coverage on Linux and macOS

Additional context

The problem goes away after removing the line <ExcludeByAttribute>GeneratedCodeAttribute,CompilerGeneratedAttribute</ExcludeByAttribute> from the .runsettings file. Another workaround is to add [ExcludeFromCodeCoverage] on the NativeMethods class.

bart-vmware avatar Jun 30 '25 13:06 bart-vmware

coverlet supports only .NET projects. Please explain NativeMethods class?

Coverlet is a cross platform code coverage framework for .NET, with support for line, branch and method coverage. It works with .NET Framework on Windows and .NET Core on all supported platforms.

Bertk avatar Jul 20 '25 07:07 Bertk

Well, just take a look in the zip. It's all .NET only.

bart-vmware avatar Jul 21 '25 08:07 bart-vmware

OK. I used your repo and there was no crash using .NET 10 preview. I used NET SDK 8.0.412 with global.json file and was not able to reproduce the crash.

By the way, I did not used the '.git' folder from the zip file.

C:\GitHub\coverlet-issue-1762>git clean -xdf
Removing ClassLibrary1/bin/
Removing ClassLibrary1/obj/
Removing Logs/
Removing TestProject1/TestResults/
Removing TestProject1/bin/
Removing TestProject1/obj/
Removing global.json

C:\GitHub\coverlet-issue-1762>dotnet test --configuration Release --collect "XPlat Code Coverage" --logger trx --settings coverlet.runsettings -verbosity:diagnostic -noconsolelogger --diag:Logs/log.txt
C:\Program Files\dotnet\sdk\8.0.412\MSBuild.dll -noconsolelogger -nologo -maxcpucount -nodereuse:false -property:VSTestSetting=C:\GitHub\coverlet-issue-1762\coverlet.runsettings -property:VSTestLogger=trx -property:VSTestDiag=C:\GitHub\coverlet-issue-1762\Logs\log.txt -property:VSTestCollect=XPlat Code Coverage -property:Configuration=Release -property:VSTestArtifactsProcessingMode=collect -property:VSTestSessionCorrelationId=14884_10797cd9-9eee-4e6c-8690-8151d31bb728 -restore -target:VSTest -verbosity:m -verbosity:diagnostic .\CoverletCrashDemo.sln
Test run for C:\GitHub\coverlet-issue-1762\TestProject1\bin\Release\net8.0\TestProject1.dll (.NETCoreApp,Version=v8.0)
VSTest version 17.11.1-release-24455-02 (x64)

Starting test execution, please wait...
Logging Vstest Diagnostics in file: C:\GitHub\coverlet-issue-1762\Logs\log.txt
A total of 1 test files matched the specified pattern.
Results File: C:\GitHub\coverlet-issue-1762\TestProject1\TestResults\bertk_LAPTOP-BERT_2025-07-21_11_10_41.trx

Passed!  - Failed:     0, Passed:     1, Skipped:     0, Total:     1, Duration: 1 ms - TestProject1.dll (net8.0)

Attachments:
  C:\GitHub\coverlet-issue-1762\TestProject1\TestResults\82106b83-e8b3-4309-8ed4-3722a2e7d169\coverage.opencover.xml

coverage.opencover.zip

My system installed .NET SDKs:


C:\GitHub\coverlet-issue-1762>dotnet --list-sdks
6.0.428 [C:\Program Files\dotnet\sdk]
8.0.412 [C:\Program Files\dotnet\sdk]
9.0.302 [C:\Program Files\dotnet\sdk]
10.0.100-preview.5.25277.114 [C:\Program Files\dotnet\sdk]

Please provide test diagnostic log files from your environment .

Bertk avatar Jul 21 '25 09:07 Bertk

Just to clarify, your log files don't contain a NullReferenceException, and a non-empty coverage report is being produced?

bart-vmware avatar Jul 21 '25 11:07 bart-vmware

The problematic error message is System.Reflection.ReflectionTypeLoadException using coverlet.collector for this scenario and a newer version of coverlet.collector is not available.

TpTrace Warning: 0 : 18412, 1, 2025/07/22, 09:10:44.586, 1677527465289, vstest.console.dll, TestPluginDiscoverer: Failed to get types from assembly 'Microsoft.Diagnostics.NETCore.Client, Version=0.2.12.27814, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. Error: System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types.
System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Bcl.AsyncInterfaces, Version=6.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the file specified.

The coverlet.msbuild package can be used instead.

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>

    <IsPackable>false</IsPackable>
    <IsTestProject>true</IsTestProject>
    <IncludeSymbols>True</IncludeSymbols>
    <SymbolPackageFormat>snupkg</SymbolPackageFormat>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="coverlet.msbuild" Version="6.0.4">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers</IncludeAssets>
    </PackageReference>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.14.1" />
    <PackageReference Include="xunit" Version="2.9.3" />
    <PackageReference Include="xunit.runner.visualstudio" Version="3.1.1">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
  </ItemGroup>

  <ItemGroup>
    <ProjectReference Include="..\ClassLibrary1\ClassLibrary1.csproj" />
  </ItemGroup>

  <ItemGroup>
    <Using Include="Xunit" />
  </ItemGroup>

</Project>

C:\GitHub\coverlet-issue-1762>dotnet test TestProject1\TestProject1.csproj /p:CollectCoverage=true -c release
Restore complete (0,5s)
You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy
  ClassLibrary1 succeeded (0,3s) → ClassLibrary1\bin\release\net8.0\ClassLibrary1.dll
  TestProject1 succeeded (0,5s) → TestProject1\bin\release\net8.0\TestProject1.dll
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v3.1.1+bf6400fd51 (64-bit .NET 8.0.18)
[xUnit.net 00:00:00.09]   Discovering: TestProject1
[xUnit.net 00:00:00.12]   Discovered:  TestProject1
[xUnit.net 00:00:00.14]   Starting:    TestProject1
[xUnit.net 00:00:00.17]   Finished:    TestProject1
  TestProject1                                                                            GenerateCoverageResult (2,1s)

+---------------+--------+--------+--------+
| Module        | Line   | Branch | Method |
+---------------+--------+--------+--------+
| ClassLibrary1 | 72.72% | 50%    | 40%    |
+---------------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 72.72% | 50%    | 40%    |
+---------+--------+--------+--------+
| Average | 72.72% | 50%    | 40%    |
+---------+--------+--------+--------+
  TestProject1 test succeeded (2,1s)

Test summary: total: 1; failed: 0; succeeded: 1; skipped: 0; duration: 1,0s
Build succeeded in 4,3s

Bertk avatar Jul 22 '25 07:07 Bertk

Thanks for investigating. We have a workaround in place; we'll wait for the next version.

bart-vmware avatar Jul 22 '25 14:07 bart-vmware

Please use the current nightly build and share the results in this issue.

coverlet.collector 8.0.0-preview.15.g893dcc8fb6

Bertk avatar Nov 17 '25 06:11 Bertk

Added a nuget.config:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <clear />
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" />
    <add key="coverletNightly" value="https://pkgs.dev.azure.com/tonerdo/coverlet/_packaging/coverlet-nightly/nuget/v3/index.json" />
  </packageSources>
</configuration>

And updated the project reference to:

<PackageReference Include="coverlet.collector" Version="8.0.0-preview.15.g893dcc8fb6">
  <PrivateAssets>all</PrivateAssets>
  <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
</PackageReference>

Still the same crash:

TpTrace Warning: 0 : 29396, 2, 2025/11/18, 13:26:15.158, 27975718991, datacollector.dll, [coverlet]Unable to instrument module: C:\Downloads\CoverletCrashDemo\TestProject1\bin\Release\net8.0\ClassLibrary1.dll
System.NullReferenceException: Object reference not set to an instance of an object.
   at Coverlet.Core.Instrumentation.Instrumenter.CollectLambdaMethodsInsideLocalFunction(MethodDefinition methodDefinition)+MoveNext() in /_/src/coverlet.core/Instrumentation/Instrumenter.cs:line 834
   at System.Collections.Generic.List`1.AddRange(IEnumerable`1 collection)
   at Coverlet.Core.Instrumentation.Instrumenter.InstrumentType(TypeDefinition type) in /_/src/coverlet.core/Instrumentation/Instrumenter.cs:line 497
   at Coverlet.Core.Instrumentation.Instrumenter.InstrumentModule() in /_/src/coverlet.core/Instrumentation/Instrumenter.cs:line 243
   at Coverlet.Core.Instrumentation.Instrumenter.Instrument() in /_/src/coverlet.core/Instrumentation/Instrumenter.cs:line 147
   at Coverlet.Core.Coverage.PrepareModules() in /_/src/coverlet.core/Coverage.cs:line 142

Are you not getting that with the minimal repro steps in the issue description?

bart-vmware avatar Nov 18 '25 12:11 bart-vmware

I spend some time for analysis but did not look further into .runsettings file. Usually I am not using this configuration method.

C:\GitHub\coverlet-issue-1762>dir /s /B
C:\GitHub\coverlet-issue-1762\.gitignore
C:\GitHub\coverlet-issue-1762\ClassLibrary1
C:\GitHub\coverlet-issue-1762\coverlet.runsettings
C:\GitHub\coverlet-issue-1762\CoverletCrashDemo.sln
C:\GitHub\coverlet-issue-1762\global.json
C:\GitHub\coverlet-issue-1762\nuget.config
C:\GitHub\coverlet-issue-1762\TestProject1
C:\GitHub\coverlet-issue-1762\ClassLibrary1\ClassLibrary1.csproj
C:\GitHub\coverlet-issue-1762\ClassLibrary1\NetworkShareWrapper.cs
C:\GitHub\coverlet-issue-1762\TestProject1\TestProject1.csproj
C:\GitHub\coverlet-issue-1762\TestProject1\UnitTest1.cs
C:\GitHub\coverlet-issue-1762\dotnet build-server shutdown
C:\GitHub\coverlet-issue-1762\dotnet test --configuration Release --collect "XPlat Code Coverage" --logger trx --settings coverlet.runsettings -verbosity:diagnostic -noconsolelogger --diag:Logs/log.txt
C:\Program Files\dotnet\sdk\8.0.416\MSBuild.dll -noconsolelogger -nologo -maxcpucount -nodereuse:false -property:VSTestSetting=C:\GitHub\coverlet-issue-1762\coverlet.runsettings -property:VSTestLogger=trx -property:VSTestDiag=C:\GitHub\coverlet-issue-1762\Logs\log.txt -property:VSTestCollect=XPlat Code Coverage -property:Configuration=Release -property:VSTestArtifactsProcessingMode=collect -property:VSTestSessionCorrelationId=22564_9819e50b-e56c-4e9b-aaf4-ed8ee6936451 -restore -target:VSTest -verbosity:m -verbosity:diagnostic .\CoverletCrashDemo.sln
Test run for C:\GitHub\coverlet-issue-1762\TestProject1\bin\Release\net8.0\TestProject1.dll (.NETCoreApp,Version=v8.0)
VSTest version 17.11.1-release-24455-02 (x64)

Starting test execution, please wait...
Logging Vstest Diagnostics in file: C:\GitHub\coverlet-issue-1762\Logs\log.txt
A total of 1 test files matched the specified pattern.
Results File: C:\GitHub\coverlet-issue-1762\TestProject1\TestResults\bertk_LAPTOP-BERT_2025-11-19_08_01_29.trx

Passed!  - Failed:     0, Passed:     1, Skipped:     0, Total:     1, Duration: 1 ms - TestProject1.dll (net8.0)

Attachments:
  C:\GitHub\coverlet-issue-1762\TestProject1\TestResults\9c7611b0-faef-4c4a-9940-e3247a446be5\coverage.opencover.xml

C:\GitHub\coverlet-issue-1762>C:\GitHub\coverlet-issue-1762\TestProject1\TestResults\9c7611b0-faef-4c4a-9940-e3247a446be5\coverage.opencover.xml

The code coverage file is empty but running test without coverlet.runsettings file will create a valid code coverage XML file. I did not find the reason for this behavior.

C:\GitHub\coverlet-issue-1762>dotnet dotnet build-server shutdown
C:\GitHub\coverlet-issue-1762>dotnet test --configuration Release --collect "XPlat Code Coverage" --logger trx -verbosity:diagnostic -noconsolelogger --diag:Logs/log.txt -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=opencover
C:\Program Files\dotnet\sdk\8.0.416\MSBuild.dll -noconsolelogger -nologo -maxcpucount -nodereuse:false -property:VSTestLogger=trx -property:VSTestDiag=C:\GitHub\coverlet-issue-1762\Logs\log.txt -property:VSTestCollect=XPlat Code Coverage -property:Configuration=Release -property:VSTestCLIRunSettings=DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=opencover -property:VSTestArtifactsProcessingMode=collect -property:VSTestSessionCorrelationId=9804_2310d3cf-3a43-41b6-abdf-45ffe25985ac -restore -target:VSTest -verbosity:m -verbosity:diagnostic .\CoverletCrashDemo.sln
Test run for C:\GitHub\coverlet-issue-1762\TestProject1\bin\Release\net8.0\TestProject1.dll (.NETCoreApp,Version=v8.0)
VSTest version 17.11.1-release-24455-02 (x64)

Starting test execution, please wait...
Logging Vstest Diagnostics in file: C:\GitHub\coverlet-issue-1762\Logs\log.txt
A total of 1 test files matched the specified pattern.
Results File: C:\GitHub\coverlet-issue-1762\TestProject1\TestResults\bertk_LAPTOP-BERT_2025-11-19_08_35_48.trx

Passed!  - Failed:     0, Passed:     1, Skipped:     0, Total:     1, Duration: 1 ms - TestProject1.dll (net8.0)

Attachments:
  C:\GitHub\coverlet-issue-1762\TestProject1\TestResults\58f6ab4e-025d-4d45-830e-b26f891c4eb9\coverage.opencover.xml

C:\GitHub\coverlet-issue-1762>

Bertk avatar Nov 19 '25 08:11 Bertk