Question: Code coverage and artifact build

[JakubLinhart]

I have an integration test suit and I parallelize its execution on mutliple test agents so it is much faster. Building solution on each test agent makes parallelization much less effective since the solution build itself takes considerable time (~10 minutes). The idea is to publish binary artifacts from one build agent and reuse these artifacts on all test agents.

It works only if binary artifacts include bin and obj of all projects in the solution. The size of all binary files is unfortunatelly more than 3 GB and downloading them to build agent takes nearly the same time as building the solution on each test agent which makes the approach useless.

It is enough to publish bin and obj (~300 MB) of all test projects to run all integration tests since it contains all required dll and pdb. But it is not enough for code coverage - resulting coverage.cobertura.xml is empty.

The good part is that working copy path is the same on all build agents but also doesn't seem to be enough. Also deterministic build (as described in this example) seems to be not enough and coverlet still produces empty coverage.cobertura.xml.

Please, is there any way how to run tests and collect code coverage just from bin/obj files of test projects without the need of bin/obj from all solution projects?

EDIT:

Source code is available on all build and test agents on the same path.

[petli]

I don't think you need anything but the bin directories of the test projects, but you may be hit by what #1164 describes where the source code must be available when running the test. Currently there's no way around it, but that issue suggests a parameter to give you control for this kind of situations.

[JakubLinhart]

@petli thanks. Source code is available on all build and test agents always on the same path. https://github.com/coverlet-coverage/coverlet/issues/1271 seems to be related to this issue. I will try to run coverlet as a command tool.

[MarcoRossignoli]

@JakubLinhart can you enable logs and check the reason why modules are not instrumented? https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/Troubleshooting.md

[JakubLinhart]

@MarcoRossignoli thanks for your suggestion. I created a minimal example project. Diag log files are here, here and here. I can't find anything interesting in these logs.

It seems like the problem shows up only when ClassLibrary1\obj\Debug\netstandard2.1\ClassLibrary1.pdb is missing, otherwise, everything is OK. You can run nocoverage.cmd batch to simulate the situation on build and test agents.

[MarcoRossignoli]

We need the pdb for the instrumentation, we use that informations to emit probes and to have mapping between code and IL. Why is missing or not found?

[JakubLinhart]

It is missing in ClassLibrary1 project but ClassLibrary1.pdb is still there in TestProject1: TestProject1\bin\Debug\net6.0\ClassLibrary1.pdb. So the question is why coverlet uses a pdb from class library project (ClassLibrary1\obj\Debug\netstandard2.1\ClassLibrary1.pdb) and an assembly from test project (TestProject1\bin\Debug\net6.0\ClassLibrary1.dll).

[JakubLinhart]

But to the question of why TestProject1\bin\Debug\net6.0\ClassLibrary1.pdb is missing: I don't want to copy bin/obj of all projects from build agent to integration test agents since they are huge (several GBs). I would like to copy bin/obj of the integration tests project only.

[MarcoRossignoli]

what is your workflow? are you publishing and after running the tests?

Unfortunately coverlet cannot instrument without pdbs file like other instrumentation method we need to know the sequence points to map probes/sources, so usually the user do the publish(where pdbs are close to the dlls) and run test or run tests directly on sln or proj where pdbs are present.

[JakubLinhart]

Yes, publishing from a single build agent and then running tests on multiple test agents.

I understand that PDB files are needed for the instrumentation. The PDB file is available on all test agents actually: TestProject1\bin\Debug\net6.0\ClassLibrary1.pdb.

The question is why instrumentation doesn't work when PDB is not available at this path: ClassLibrary1\obj\Debug\netstandard2.1\ClassLibrary1.pdb. Or in other words: Why coverlet cannot use PDB copied to test project output path TestProject1\bin\Debug\net6.0\ClassLibrary1.pdb.

[github-actions[bot]]

This issue is stale because it has been open for 3 months with no activity.