[FR] Expose the computed config programmatically

[webknjaz]

So I've been playing with setting up gcovr to measure coverage for Python modules written in pure C (not Cython). I got in working as a PoC in the dev environment, however I've faced challenges integrating it with the CI (https://github.com/aio-libs/multidict/pull/1228). There's a lot of moving pieces and path mapping complexity that I've mostly overcome. However, it turned out that gcovr invokes the gcov program of GCC to analyze the *.gcno and *.gcda files.

*.gcno is produced when invoking gcc when it's set up to produce tracing-related data (so during the compilation stage) and *.gcda is written to disk when the C-extension is actually executed (as in when running pytest that imports the compiled module).

After that, a carefully composed gcovr command is able to produce HTML, XML and other coverage outputs.

This all works well in a dev env with something like an editable install. However, in CI we have cibuildwheel jobs that produce manylinux wheels in containers, and following jobs install and test those wheels. I've able to make it so that *.gcno ends up in site-packages/ as included package data and my pytest setup is able to properly transfer that into the CWD and set environment variables that GCOV requires to write *.gcda into the same folder.

The problem is that the gcovr invocation in CI it's an error when running gcov (./_multidict.gcda:version 'B42 ', prefer version 'B43*' and a return code of 3).

It turns out that I have to use the same toolchain for analyzing coverage as was used to compile C-extensions: https://stackoverflow.com/a/60347862/595220.

So my idea is that I'd extract the appropriate manylinux container value from cibuildwheel and run gcovr inside it, in the test jobs, after running pytest. I've validated locally that this would work by running the same container image interactively. Is it possible to expose this (or even the entire config computed from the file, CLI args and env vars — this would be more generic)? Perhaps a subcommand would be useful, showing what container is selected based on the wheel name? Is the config somehow available from within a cibuildwheel run? Any other ideas?

[joerick]

I guess it wouldn't be too much work to create a subcommand like --compute-config IDENTIFIER that returns a human-readable/JSON representation of the config. But, I wonder how generally useful it might be? It feels like a little bit of a narrow use-case :)

The other thing you could try is to get the options yourself using compute_options? It's an internal API but hasn't changed in a while...

from cibuildwheel.options import compute_options

options = compute_options(
  platform='linux', 
  command_line_arguments=CommandLineArguments.defaults(), 
  env=os.environ
)
build_options = options.build_options('cp310-whatever-identifier')
print(build_options.manylinux_images['x86_64'])

[webknjaz]

Thanks for the ideas!

Upon reflection, I have another one — what if there was a way to ask cibuildwheel to run something in the build env. The problem is, though, that we'd need to trace that env back from the wheel. I wonder if cibuildwheel could expose/write a mapping of the produced wheels to their build envs that would be consumable by another cibuildwheel invocation that would be able to run a given command in the same env but skip building everything made by the first one. Though, even if the container refs were exposed somehow, that'd improve what's possible greatly.

I understand that the use case is rather narrow. But I feel like this is a chicken-egg problem, too. It's non-trivial to set up gcovr integration, and requires the deprecated invocation of setup.py to work most of the time. I'm working on making it functional with a regular pip install. Somebody has to improve the DX, after all. When it's easier to set up, I don't see why this use case should remain niche. That's why I'm seeking advice and hoping to produce something other people would be able to reproduce. What we have now is rather fragile, and so I can see how that discourages devs from measuring coverage in C.

[joerick]

For this second idea, perhaps I'm not following but I wonder if you could append this to your test-command? So you do pytest <something> && gcovr ?

[webknjaz]

@joerick the challenge is that I don't want to run tests in the build jobs in CI. I maintain clear separation. And so the test jobs might run testing in environments different from the build env (external to cibuildwheel). But the gcovr run for post-testing coverage analysis/conversion is what needs the original build tool chain.

I have a build matrix with cibuildwheel jobs that's its own stage. And then, in the test jobs, the dists are downloaded and one compatible wheel is selected and pip installed. So I know the wheel being tested. I just need to somehow go from that wheel to the actual build env (which would be the container reference, I imagine).

Though, I feel like we could benefit from having something like cibuildwheel --select=cp313-manylinux_x86_64 exec "gcovr" that could take us into a prepared build environment but instead of building would execute whatever. Similar to podman -it exec "cmd" (or docker for that matter).

Now that I think about it, this could be a good addition to cibuildwheel's interactive troubleshooting capabilities, too.