python coverage not supported with `--branch`

[dgw]

The SQL query below used to parse .coverage for Python projects returns 0 results if coverage was run with --branch on the CLI or with branch = True in .coveragerc.

https://github.com/coverallsapp/coverage-reporter/blob/fd443e47d027db1dcf5671e6f4e18138a06deb50/src/coverage_reporter/parsers/coveragepy_parser.cr#L10-L14

With this argument/setting, coverage seems not to add any rows in the line_bits table, and coverage-reporter will therefore find that there is "nothing to report" using the above query; only the arc table is populated with --branch, and it isn't queried.

Omitting the --branch argument or removing/commenting out the branch = True configuration setting lets coverage-reporter find results in the .coverage file, with the obvious caveat that only statement coverage data is reported.

Note that without --branch, the arc table is now empty, and line_bits is populated.

---

I attempted to work around this by running coverage lcov prior to the coverallsapp/github-action@v2 step, and it appears to work, but doesn't seem ideal. With zero code changes, the project's coverage jumped by 0.6% and spot-checking a few places showed a lot of unexpected branch coverage annotations on Coveralls. For example, partially covered branches I checked are now shown as fully covered, and sometimes branch-coverage annotations appear in the Coveralls UI where there is no branch at all:

I'm unsure yet whether coverage outputs bad lcov format when branch tracking is enabled, or if coveralls-reporter isn't parsing it correctly. At this point I've only managed to figure out that something isn't working as expected. 😅

[mrexox]

Hey! Thank you for reporting the issue. Python coverage calculation from .coverage file is very basic and doesn't support branch coverage. This is marked in the README (beta support). To use get more stable coverage please convert .coverage to XML format (see instructions).

[dgw]

Hi @mrexox! Good pointer on the XML reports; that fixed a lot of the issues, and it's clearly on me for not reading the documentation closely enough. 😅

After switching to XML reports, a new display issue on Coveralls was easily solved by tweaking coverage's configuration to include paths instead of specifying source (compare file trees before and after the config change); that change also fixed Coveralls not being able to fetch the source files.

---

However, even after figuring out the correct format and configuration, coverage upload with the new reporter is still a step down from uploading with the coveralls PyPI package. Partially-covered branches are rendered as yellow with a '!' when uploaded with the older tool, but show as green with this newer one. Here is another example that shows as green and displays an "All branches covered" tooltip despite the nested code not being covered (older version w/yellow partial-coverage display).

Going beyond just the color-coding, branch-coverage tooltips are also less useful with the newer reporter. Compare this block in a build uploaded by coveralls from PyPI:

…against the same location as uploaded by this newer binary using coverage xml output:

In this particular case, the if statement has two conditions (if one_var and a_second_var:), but all if statements I've checked with partial coverage display three lines like this. Besides the condition count mismatch, it's objectively less useful to show "branch 0 missed" than the old output with line numbers of the missed branch targets.

---

That's enough nitpicking from me for one day, I'm sure. Better support for Python coverage reports is undoubtedly in the works, and we'll keep an eye out for developments. For now, though, as long as the older tool still works and provides more readable reports, I think we'll put our migration (sopel-irc/sopel#2551) to the newer binary on hold.

[littleforest]

Closing this as python parser now converts .coverage to XML with coverage xml and uses the Cobertura parser. See #122.

[dgw]

I ran a quick test of this using the following workflow steps:

      - name: Run pytest
        run: make test_norecord
      - name: Generate coverage report
        run: coverage report
      - name: Upload coverage report to coveralls.io
        uses: coverallsapp/github-action@v2
        with:
          flag-name: ${{ matrix.python-version }}
          parallel: true

In the resulting Coveralls build, other than our build configuration not yet aligning with the ideal for avoiding "drifted build" warnings, branch coverage still is not being reported as expected after coverallsapp/github-action uses v0.6.10 of the reporter, released earlier today, to upload the .coverage file results.

Branch coverage reported using coverage xml / file: coverage.xml in the workflow is also not being reported as expected; the results are very similar to my last comment with screenshots. Our master branch using an older Coveralls tool to upload the results remains unaffected; partial branch coverage still shows the branch point in yellow as expected.

Unless I've configured our test job wrong (see recent commits to sopel-irc/sopel#2551), it doesn't seem to me that this issue is actually resolved yet. 😕

[afinetooth]

Thanks, @dgw. We will have another look at this.

The question is:

Why were previous conditional statement (with branch coverage) showing partial coverage (yellow) when their child branches had misses (correct), while, now, those same conditionals, whose child branches still have misses, now show as fully covered (green)?

Example provide by @dgw: https://coveralls.io/builds/63927960/source?filename=sopel%2Fbot.py#L888

Cc: @littleforest @mike-burns