Add --include and --exclude options to auditwheel repair

[rossant]

This is a quick fix to solve an issue I have with my project and for which auditwheel seemed to help. I don't know if this approach is suitable to other users of auditwheel.

I develop a C graphics library (datoviz) that depends on Vulkan and comes with Cython bindings. I'm trying to build a pip wheel for Linux (at least Ubuntu 20.04, that would be a start).

1. As a first step, I compile the C project and get a libdatoviz.so shared library.
2. Then, I build the Cython library, which has a dynamic dependency to libdatoviz.
3. I'd like to bundle both the Cython extension module, and libdatoviz, in the same wheel.
4. I tried to use auditwheel repair. It included dozens of other dependencies in the wheel, including libvulkan and many graphics-related dependent libraries. The compiled wheel installs properly, but it doesn't work properly (there are issues related to windowing, GPU access, etc). Related: https://github.com/pypa/auditwheel/issues/241
5. I made some changes to auditwheel (this pull request) to exclude libvulkan and a few other libraries. Some issues were fixed, but not others.
6. As another approach, I tried to bundle just libdatoviz, and no other library in the wheel. That seems to work, at least on my computer. Chances are that the wheel may not work on other Linux-based operating systems though.

Any help would be appreciated, regarding either this pull request, and/or a way to solve the issue I'm facing. Thanks!

[codecov[bot]]

Codecov Report

Merging #310 (607b8f7) into master (e5d99f0) will decrease coverage by 0.66%. The diff coverage is 50.00%.

@@            Coverage Diff             @@
##           master     #310      +/-   ##
==========================================
- Coverage   91.28%   90.62%   -0.67%     
==========================================
  Files          20       20              
  Lines        1102     1120      +18     
  Branches      237      242       +5     
==========================================
+ Hits         1006     1015       +9     
- Misses         54       60       +6     
- Partials       42       45       +3     

Impacted Files	Coverage Δ
auditwheel/repair.py	80.59% <43.75%> (-5.00%)	:arrow_down:
auditwheel/main_repair.py	90.32% <100.00%> (+0.32%)	:arrow_up:

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update e5d99f0...607b8f7. Read the comment docs.

[mayeut]

Thanks for your contribution @rossant

Having an --exclude option, while dangerous, allows to work around those kind of issues in auditwheel (e.g. libvulkan but probably also libOpenCL, libOpenVX...)

Fixing those kind of issues requires a considerable amount of time related to the analysis of what's available on which distro, will it be compatible, is there any kind of commitment from those libraries on ABI stability for the foreseeable future...

This would allow projects using this option to do the analysis for their specific need. If the analysis is complete and allows the libraries to be whitelisted per PEP 600, then it might be better for the community to get auditwheel to update its policies.

I'd rather not have the --include option present but only the --exclude one as it is explicit what's not being included.

Having the --exclude option would help mitigate #241, #250, #152 (and thus #161) while waiting for a more generic "foolproof" fix.

@lkollar, any thoughts ?

[rossant]

Status update:

• I repaired my wheel with --plat linux_x86_64 but then I realized this wouldn't work with PyPI. Of course I wasn't able to build a manylinux wheel on my development machine.
• I'm investigating the Docker manylinux way now. Using the manylinux_2_24_x86_64 might work, but it's tricky with the others since the Vulkan SDK doesn't seem to support old distros (see https://vulkan.lunarg.com/issue/view/5c5feabb66315107d5ab4753#comment-5ec66580670a0526d975a86e).
• I ran into many sorts of problems using --exclude, so I ended up using only --include. I only include my library, nothing else. I don't expect this to work on many Linux platforms. My main goal at the moment with my project is supporting at least Ubuntu 20.04 (my development machine) with a precompiled wheel package on PyPI. I'm not sure what the PyPI policy is regarding the manylinux wheels that fail on some of the Linux distros officially supported by PyPI.
• A similar approach seems to work on macOS with delocate (using --include) but I need to test it more thoroughly.
• My build process will use my fork of auditwheel for now, so in the short term I don't depend on this PR being merged..

[mayeut]

@rossant, If you expect your wheel to only be ubuntu 20.04 compliant, then it's not a good idea to upload it to PyPI. It will only confuse end-users to see a manylinux wheel that doesn't work and will end up with reports to your repo tracker and/or PyPI issue tracker. In this case, your better of publishing your specific wheel as a release artifact on GitHub for example and update your docs to mention how to install it. see also https://github.com/pypa/warehouse/issues/5420

[rossant]

@mayeut makes sense, thanks. Are there guidelines somewhere about how to properly test a manylinux wheel on a sufficiently wide range of distributions to ensure it reaches the bar for PyPI?

[mayeut]

Are there guidelines somewhere about how to properly test a manylinux wheel on a sufficiently wide range of distributions to ensure it reaches the bar for PyPI?

There's no such guidelines to my knowledge. Personal opinion: for simple packages, running tests in the manylinux docker is almost always enough. I'd say add to that a recent distro if you have "simple" external dependencies. When you have not so simple dependencies it's a bit more complex I guess.

PEP 600 only state (short extract, I advise to look at full spec):

the wheel's creator promises that the wheel will work on any mainstream Linux distro that uses glibc version ${GLIBCMAJOR}.${GLIBCMINOR} or later ... The word "mainstream" is intentionally somewhat vague, and should be interpreted expansively. The goal is to rule out weird homebrew Linux systems; generally any distro you've actually heard of should be considered "mainstream".

For a manylinux_2_31 wheel (ubuntu 20.04 uses glibc 2.31), given the dependencies you want to exclude, I'd look at the non EOLed distros in https://github.com/mayeut/pep600_compliance#distro-compatibility starting at manylinux_2_31. In theory, if one of those distros can't run your wheel, it should not be eligible for manylinux distribution (I would add an exception on photon which is a headless only distro, given you're using the GPU, this distro probably does not apply).

[martinRenou]

Having an --exclude option, while dangerous, allows to work around those kind of issues in auditwheel (e.g. libvulkan but probably also libOpenCL, libOpenVX...)

I'd also be interested in this change being included!

@rossant are you planning on updating your PR? If time is not on your side I can open another PR myself adding only the --exclude option as suggested by @mayeut (while giving you credit for the commit)

[rossant]

@rossant are you planning on updating your PR? If time is not on your side I can open another PR myself adding only the --exclude option as suggested by @mayeut (while giving you credit for the commit)

I'd definitely appreciate some help! I won't have time to update my PR in the short term

[martinRenou]

Ok :) I'll push a PR soonish

[martinRenou]

See https://github.com/pypa/auditwheel/pull/368

[mayeut]

closing in favor of #368