Consider partitioning Coverage reports by interpreter constraints

[Eric-Arellano]

Problem

The version of coverage you use at report time matters for being able to parse the syntax of your code. Coverage parses all the files when generating the report.

We generate one report for your whole repo, meaning you can’t have Py2-only code and Py3-only code.

Workaround

The user can set ignore_errors = True in their coveragerc file to ignore the files that can't be parsed. Those files will simply be left off from the report.

Possible solution

Have Pants partition at report-time and generate a report per partition. Note that we already partition at data generation time, this is at report time.

Complication: one report often expected

Tools like Coveralls often expect only one report. We may break workflows if there are multiple files.

John suggested that we can generate a global .coverage raw file still, and then also create all the different partitioned reports on top of it.

Complication: name collisions

When generating a file, we'd need to mix in the interpreter constraints for that partition to avoid name collisions.

One tricky detail is how to represent complex constraints like ['CPython>=3.5,<4', 'CPython<=2.7']. We should probably not be using <, >, !, and = in the file name. A textual representation like coverage_cpython_ge_3.5_lt_4_or_cpython_le_2.7 is very verbose.

[jsirois]

Instead of representing constraints why not partition on the actual interpreter used to generate the .coverage data file. Then you have a single interpreter to work with instead of constraints when trying to concoct a unique report name.

[github-actions[bot]]

This issue has been open for over one year without activity and is not labeled as a bug. It has been labeled as stale to invite any further updates. If you can confirm whether the issue is still applicable to the latest version of Pants, appears in other contexts, or its priority has changed, please let us know. Please feel free to close this issue if it is no longer relevant.