pip should rebuild the cython module when it is missing in the downloaded wheel

[yurivict]

What's the problem this feature will solve?

Feature or enhancement

Proposal:

The wheel for the GPy package lacks cython-compiled binary modules on the ARM64 architecture.

For example, the choleskies_cython cython submodule is only available for the x86_64 architecture:

[yuri@yv /tmp/GPy-wheel/GPy]$ find .  | grep choleskies
./util/choleskies_cython.cpython-311-x86_64-linux-gnu.so
./util/choleskies_cython.c
./util/choleskies.py
./util/choleskies_cython.pyx

In Python-3.11 pip installs a broken GPy module on the ARM64 architecture.

Describe the solution you'd like

When the binary should be present (the corresponding *.c and *.pyx files are present) but is missing, pip should rebuild this binary during the pip install xx command.

Alternative Solutions

n/a

Additional context

n/a

Code of Conduct

• [x] I agree to follow the PSF Code of Conduct.

[notatallshaw]

Thanks for raising this issues. However I don't think pip would start doing non-standard installation behavior independently of the rest of the Python packaging ecosystem. Pip attempts to be a standards bearer so that tools only need implement the standards and not just look at how pip does something.

As such you should propose new installation behaviors in https://discuss.python.org/c/packaging/14.

[RonnyPfannschmidt]

If the wheel ships cpython stuff without platform tags the package is wrongly uploaded

The project should fix it

Pip is not responsible for invalid uploads

[pfmoore]

I don't think pip would start doing non-standard installation behavior

More specifically, it's an important security feature of wheels that no build takes place when installing a wheel. If the GPy project is shipping wheels that are marked as being compatible with ARM64, but which are missing essential binaries, that sounds more like a packaging issue for the GPy project to address.

[RonnyPfannschmidt]

The packaging tooling of that package is outdated and incorrect

They need to fix it

[yurivict]

I agree that GPy is outdated and has a missing ARM64 binary.

However, pip shouldn't just create a broken installation.

There are a lot of possible architectures. For example, someone might need to build for MIPS, or for I386. The package should support this by default because both Python and C/C++ code is platform-independent. But package authors can't possibly anticipate what architecture would users build for and cover all architectures.

Python tooling is able to build missing binaries, and it should do this or at least suggest this.

This is why I think that pip shouldn't create a broken installation just because the binary for some ARCHX is missing. It should either fail and suggest the user how to rebuild the binary, or it should rebuild it. I am not sure what's the right behavior here, but it should at least warn the user.

Possible behaviors:

1. Error message: The GPy wheel is missing a binary for the architecture XXX. Please rebuild the binary using "python -m build --wheel ." in the package directory, or install a broken package by setting the PIP_IGNORE_MISSING_BINARIES=yes environment variable.
2. If the wheel is broken, attempt to download the source tarball and build it.

[pfmoore]

pip has no way of knowing which architectures need binaries. If you want to build from source, you can use pip install --no-binary=GPy Gpy. From the sound of it, though, you may find that GPy doesn't build cleanly anyway, and you'll need to do some more work to build it.

[notatallshaw]

High quality PRs welcome for improving error messages, if there is some way to detect a broken install here that would be helpful.

But a wheel is "just" a zip file that is extracted to a specific location with specific metadata. I don't see any obvious way pip would know that the installation is broken when tbe zip file is consistent and matches the metadata.

Your suggestions are too broad, there's no rule in general a wheel will have certain binaries or what platform they are for. If you have a idea but are not able to create a high quality PR you must be provided a very specific technical description of how you think pip could detect this issue.

[ichard26]

Apologies if I'm missing something, but I don't understand what the issue is here.

It's said that the wheel for the GPy package is unsupported on ARM64, but you linked to a wheel that's explicitly tagged for the x86_64 architecture. Yes, this wheel is unsupported on the platform and that's a problem, but pip won't even consider this wheel for installation in the first place. As far as I can tell, GPy doesn't publish pure Python wheels (tagged py3-none-any) so pip—realizing that there is no compatible wheel for the ARM64 platform—will fallback to selecting the sdist and building it locally. That's one of the solutions you suggest.

Is there some other behaviour you're looking to suggest?

Yes, pip can try to offer better advice when building from source fails for non-pure projects, but that seems orthogonal to this exact issue.

[yurivict]

The best solution in this case would be to attempt to build the wheel from source. The current behavior when pip installs a broken package should be avoided. AFAIK pip builds from source when no wheel files are available.