Support for Gcov coverage in extensions

[jonathf]

What's the problem this feature will solve?

I have a Python package with a Pybind11 extension and I'd like to create coverage reports for both the Python and the C++ code. I can do this by doing the following running:

CFLAGS="-coverage" python setup.py build_ext --inplace
coverage run -m pytest ...
find src -type f -name '*.cpp' | xargs -I{} gcov -o build/temp*/src {}

This works because -coverage adds .gcno files in the build directory, which coverage recognizes and creates .gcda files. Finally gcov is used to create gcov files.

Describe the solution you'd like

Instead of using old not recommended python setup.py, I'd rather do something like:

CFLAGS="-coverage" pip install -e .

and have setup.cfg with:

[build_ext]
inplace=1

be enough to get a build directory with .gcno in it.

Alternative Solutions

Yes with old python setup.py

Additional context

Obviously this assumes gcov is installed on the system.

Code of Conduct

• [X] I agree to follow the PSF Code of Conduct.

[pfmoore]

This requires the ability to pass "configuration options" to the build backend. Pip currently doesn't support this, mostly because most build backends don't use config options.

To get this to work would need:

1. Setuptools (your build backend) to support setting the flags you need via some combinations of PEP 517 config_options values.
2. Pip to add a UI for users to pass config options to the backend.

A simple UI would be easy enough to add to pip, but it's not likely to get much priority until backends start supporting user configuration in the PEP 517 interface. So I'd suggest you initially explore how to make this work with the setuptools project, and then we can look at the front end UI in pip.

[pradyunsg]

Agreed, it would probably be a good idea to push this to setuptools, since they’d be the right folks to decide on the mechanism they wanna use to enable pip doing something like this.

[jonathf]

Okay.

Note that this is not about the copile flags per-say. Since this is a compiled component, I still need the setup.py, which I can configure to handle the CFLAGS. I added it to the example to make the example complete.

But irrespectively of how the compiler get the added flags, no gcno files are generated when compiling with pip, only through python setup.py build_ext, which as I understand is no longer recommended.

[SethMMorton]

I would like to ping this issue - as far as I can see, the only way to get gcov to successfully run for a C/C++ extension is to use python setup.py build_ext or python setup.py install, etc. However, setuptools is advising us to move away from this.

Is there some guidance on how to get equivalent behavior to python setup.py build_ext with pip install -e .?

[skirpichev]

An update. It seems, recent setuptools upgrade broke this approach to create coverage reports.

This was working til v79.0.1, see e.g. this build log. Now this require pinning: https://github.com/diofant/python-gmp/pull/155/commits/2ea82c31e391b0394f6e70aa2de599095492b7eb

I suspect this was broken by https://github.com/pypa/setuptools/issues/4955. Do we have some workarounds instead of pinning? @pfmoore ?

[pfmoore]

No idea, sorry. This isn't a pip issue, and I don't know enough about setuptools to comment from that perspective.

[skirpichev]

@jaraco (as author of #4955) does this breakage looks severe enough for you?

[jaraco]

Sorry for the delay in responding. I'm just getting to this in my backlog.

This was working til v79.0.1, see e.g. this build log. Now this require pinning: diofant/python-gmp@2ea82c3

Unfortunately, the build log has expired. Can you summarize/characterize what use case was affected? In particular, are build_ext commands affected? My instinct was that only setup.py develop was affected by the change.

@jaraco (as author of #4955) does this breakage looks severe enough for you [to revert the change]?

Maybe, but probably not. The old develop behavior has some unwanted legacy behaviors that add a lot of debt and maintenance burden to this project. It's possible that there have even been subsequent changes that depended on that removal that would also need to be reverted. I'd like to avoid that outcome if possible. I'm glad to hear that pinning has been a suitable workaround and I'd prefer to lean on that workaround until a long-term solution can be developed.

Is there some guidance on how to get equivalent behavior to python setup.py build_ext with pip install -e .?

I wonder if pip install is the right design/approach for the stated problem. On one hand, "install implies build" means that each install operation does perform the different build operations, so exposing those intermediate artifacts in some way seems an intuitive approach. On the other hand, any "install" operation does more than just build, so has potentially unwanted behaviors (altering the environment). The use of pip install -e appears to be an incidental choice that allows the build artifacts to be made available in situ instead of being copied to the site-packages directory. It's not obvious to me that an editable install would necessarily leave extension modules present alongside the source.

These observations lead me to think that maybe the Python Packaging standards don't really address this use-case. In the older/deprecated distutils/setuptools design, the different operations were separate but then composed to produce the higher-level operations (i.e. install called build, which called egg_info then build_ext, etc). The packaging ecosystem instead has two high level operations, build (to wheel or sdist) and install (editable or natural). It's not at all obvious to me that the best way to get access to the build artifacts is to perform an editable install (with special options to possibly tweak how that install is performed).

This also leads me to wonder - why is pip install -e not adequate currently? Is build_ext.inplace not honored from setup.cfg? Where do the build artifacts go under a PEP 517 build of these extension modules? Can you help create a minimal reproducer (maybe a repo) that demonstrates the issue (i.e. has an extension module that gets built in place under older Setuptools but not under pip install -e)?

[skirpichev]

Can you summarize/characterize what use case was affected? In particular, are build_ext commands affected? My instinct was that only setup.py develop was affected by the change.

My extension used setup.py develop to get coverage report, like gmpy (for now): https://github.com/aleaxit/gmpy/blob/1418b427090dd0ef851c9fd558ed9d986ac83f56/.github/workflows/pip_install_gmpy2.yml#L59-L67

Now it's impossible (statistics empty).

Maybe, but probably not.

Maybe. For me, solution was - migration from setuptools as fast as I can, even for pure-Python projects.

So, I'm not interested anymore, sorry. Maybe OP can add something. Now this looks like a feature request for setuptools. Nothing worked.

I wonder if pip install is the right design/approach for the stated problem.

I experimented with -e for setuptools, it wasn't working, I don't remember reasons.

Nowadays I use meson-python: https://github.com/diofant/python-gmp/blob/925012ce7a48c975e4357493d93681583d71f36e/.github/workflows/coverage.yml#L37-L52

Maybe this workflow helps someone.