Support coverage for `py_test`

[hwright]

bazel coverage should generate a coverage report for the python tests it runs. Ideally, it gathers the list of dependencies, and then computes coverage over that for a given test. When multiple tests are run in the same Bazel invocation, their reports should be merged into a single one.

I'm dropping this here mostly as a python-specific version of bazelbuild/bazel#1118. I know that the py_test rule is currently implemented natively in Bazel, and the implementation in rules_python just forwards there, but this is fundamentally a python issue, so this repo feels appropriate.

The general plan is to modify the python test runner within Bazel to use coverage.py to generate reports when running under coverage. I suspect it is a journey fraught with peril.

Alternatively, if it is possible to invoke coverage.py in ways other than modifying core Bazel, I'm open to suggestions.

[mattmoor]

@damienmg assigning to you to redirect appropriately.

[hwright]

fwiw, I've also been poking at this as documented here: https://groups.google.com/d/msg/bazel-discuss/LC97zBrMG84/cfVNdus-AwAJ

It's been slow going, as I'm pretty unfamiliar with Bazel internals, and haven't been able to get the ear of somebody who is.

[brandjon]

Setting P4 because this isn't a blocker for 1.0 support.

[JoshSullivan11]

Has this feature been implemented yet? If so, in what release version? It seems like coverage has been able to cover every other bit of our code repo except for all python files which is makes up most of our code base. Please let me know!

[siddharthab]

Is there a way to make bazel coverage //some_py_test not fail even if it does not collect coverage information. Currently, it fails because :lcov_merger attribute is not defined in py_test rules.

This is kind of important because we want to be able to run bazel coverage ... on a monorepo.

[sgammon]

how is it possible that coverage, python, and bazel don't work together, in the year of our lord 2020

[thundergolfer]

IIUC this problem lies in bazelbuild/bazel, and we need to work to get https://github.com/bazelbuild/bazel/issues/10660 triaged there.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days. Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!

[EnergySRE]

We at Camus Energy still want this. Python doesn't scale well without good test coverage, and coverage reports are valuable leverage.

If the author of an eventual PR is ever in San Francisco, we will buy you a taco, and tell you all about how weird it is to develop software in the energy sector. (Hopefully these words of support count as activity.)

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days. Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!

[EnergySRE]

Boop. We still desire this 🙂

[mihaigalos]

Boop. We still desire this 🙂

Then implement it, perhaps? 🙃

[adam-azarchs]

At this point upstream coveragepy has lcov support. I think the main piece that's missing is that there needs to be a way to specify a target for coveragepy in py_runtime for hermetic python toolchains.

[chrislovecnm]

@aignas do we have more work to do on this?

[sgammon]

this issue has been open for 6 years

[aignas]

I think we can safely close it now as we have docs and there are quite a few versions supporting toolchain with coverage.py enabled.

[sgammon]

@aignas I just set up a brand new project on top of rules_python, using Bazel 7 and Bzlmod, and coverage still doesn't work out of the box.

Where do you see toolchains supporting coverage?

[adam-azarchs]

https://github.com/bazelbuild/rules_python/blob/main/docs/coverage.md

[aignas]

See https://rules-python.readthedocs.io/en/latest/coverage.html