kotlinx-kover icon indicating copy to clipboard operation
kotlinx-kover copied to clipboard

[XML] Inaccurate coverage for files with same name, but from different source sets

Open IgnatBeresnev opened this issue 3 years ago • 7 comments

Describe the bug

XML report generates inaccurate coverage for files with same name, but from different source sets.

Take the following project structure as an example:

2022-11-21_13-38-18

where KoverFile.kt will contain:

  • KoverCommonClass for commonMain source set
  • KoverJvmClass for jvmMain source set.

Generated XML report will look the following way:

<?xml version="1.0" ?>
<report name="Intellij Coverage Report">
    <package name="kover/reproducer">
        <class name="kover/reproducer/KoverCommonClass" sourcefilename="KoverFile.kt">
            ...
        </class>
        <class name="kover/reproducer/KoverJvmClass" sourcefilename="KoverFile.kt">
            ...
        </class>
        <sourcefile name="KoverFile.kt">
            <line nr="3" mi="0" ci="2" mb="0" cb="0"/>
            <line nr="3" mi="0" ci="2" mb="0" cb="0"/>
            <line nr="5" mi="0" ci="4" mb="0" cb="0"/>
            <line nr="5" mi="0" ci="4" mb="0" cb="0"/>
            <counter type="INSTRUCTION" missed="0" covered="12"/>
            <counter type="BRANCH" missed="0" covered="0"/>
            <counter type="LINE" missed="0" covered="4"/>
        </sourcefile>
        ...
    </package>
    ...
</report>

As you can see, both classes lead to the same source file, the description for which duplicates line numbers. Because the code is the same in both classes, the duplicated lines are the same, but I imagine if the code is vastly different, it'll be a mess.

This leads to codecov.io ignoring such files altogether, presumably because it doesn't know how to map it to project's git structure. See codecov report. This, in turn, affects calculated coverage. Umbrella issue: https://github.com/Kotlin/kotlinx-kover/issues/16

This problem does not exists in HTML reports as it doesn't offer file-based coverage.

(additional files CommonFile and JvmFile are added to demonstrate that codecov sees Kotlin files otherwise)

Expected behavior

Not sure what could be done in this situation as it seems to be a limitation of XML report's structure, but maybe some additional kover-specific tags could be added, or some other way to differentiate between source sets.

Reproducer

Project (notice the branch): https://github.com/IgnatBeresnev/kover-reproducers/tree/mpp-same-filename

There are instructions in the README on how the report can be generated and uploaded to codecov.io

Reports Gist with XML report: https://gist.github.com/IgnatBeresnev/418ab0514b699fea7473cb2d4d275480

Environment

  • Kover Gradle Plugin version: 0.6.1
  • Gradle version: 7.4.2
  • Kotlin project type: Kotlin/Multiplatform with Common and JVM source set

IgnatBeresnev avatar Nov 21 '22 12:11 IgnatBeresnev

We will do as in JaCoCo, for summing up the values for different classes. Since the XML format does not assume that there are duplicates of files with the same name, then there is no single correct solution that other tools will work with.

shanshin avatar Nov 21 '22 18:11 shanshin

To add context to the message above, source files will have a path relative to the source set roots, similar to how it's done for classes:

<class name="kover/reproducer/JvmClass" sourcefilename="JvmFile.kt">
    ...
</class>

This will solve the problem of duplicated sourcefile lines and allow to differentiate between files from different locations. However, if both files share the same relative path, they will still be indistinguishable

IgnatBeresnev avatar Nov 21 '22 19:11 IgnatBeresnev

The 'sum up' behavior is done in agent version 1.0.688

zuevmaxim avatar Nov 22 '22 10:11 zuevmaxim

To use custom engine you need to configure it in all projects where Kover is used

for Kotlin Gradle script

kover {
    engine.set(kotlinx.kover.api.IntellijEngine("1.0.688"))
}

for Groovy

kover {
    engine = new kotlinx.kover.api.IntellijEngine("1.0.688")
}

shanshin avatar Nov 22 '22 14:11 shanshin

Relates https://github.com/codecov/uploader/issues/913

shanshin avatar Nov 22 '22 14:11 shanshin

Any news on this?

rfermontero avatar Nov 27 '23 12:11 rfermontero