colcon-core icon indicating copy to clipboard operation
colcon-core copied to clipboard

Propose colcon universal coverage result tool for python and cpp

Open Briancbn opened this issue 4 years ago • 5 comments

Past work and tickets:

  • https://github.com/colcon/colcon-core/issues/110: lcov for cpp
  • https://github.com/colcon/colcon-core/issues/349: coveragepy for python

These two all work wonders for packages that are purely cpp or python, but not for those packages that has both cpp and python package.

We can combine the two results and generate output in the following format that is available to both

  • Terminal Display
  • Cobertura xml format (which is good for codecov.io and jenkins)
  • html (Good for debugging and improving coverage)
  • json (for coveralls.io) format

There is a pycobertura, that can combine cobertura xml results. However, its API is said to be 'not stable' before the 1.0.0 version. We can either develop using its API or come out with our own ones.

With that it is much easier to

  • generate and evaluate coverage results locally,
  • implement code coverage for CI and integrate with reporting tools like codecov, coveralls, gitlab, jenkins etc.
  • provide a universal report template, when new language like Java is complete supported, we can easily add them in by supporting the format.

The first two are especially for packages that has both python and cpp packages inside

Lastly about the library used, I would suggest to use

  • gcovr instead of lcov
  • coveragepy

gcovr supports a lot more report format compared to lcov and it's python based, which can directly be implemented into colcon using its API like coveragepy

I don't have a prototype yet, but please let me know your thoughts on whether this is needed and if there are design ideas or examples I should look more at.

Briancbn avatar Jun 21 '20 06:06 Briancbn