Poetry should attempt to find both wheels and sdist during locking - across sources

[f3flight]

• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] I have searched the FAQ and general documentation and believe that my question is not already covered.

Feature Request

Consider the following scenario - separate indexes are used for wheels and sdists. User wants to prefer wheels over sdists, even if wheel is older version, but if wheel is not available, use sdist.

With pip it is done by using pip install --index-url ... --extra-index-url ... --prefer-binary ...

With poetry, there are 2 issues:

1. no prefer-binary configuration option - we cannot ask poetry to lock an older wheel if newer sdist is available.

However since in our scenario we have separate indexes for wheels, we can leverage "priority" parameter. Setting "default" for wheels index, and "supplemental" for sdist index achieves equvalent of "--prefer-binary". Of course this is not a universal solution, but in this scenario it works. But here's the second issue:

2. If poetry finds matching files in default or primary index, it does not check supplemental index at all. Why is this a problem? Because in the aforementioned scenario, poetry would only put wheels into the lock file. wheels may be bound to a certain python version. If user decides to support a newer python version, poetry install will start to fail, because it did not record sdist in the lock file, and has no compatible wheel in the lock file. Pip would install sdist in such case.

I think we should add an option to ask poetry to look for at least one instance of wheel and one instance of sdist - separately, so for each type it would separately go through default+primary, then supplemental sources. prefer-binary flag would help as well here, telling poetry which type to use to lock the version (while still trying to find both types for this selected version).

(if there's a way to lock only versions w/o locking specific files or source urls - that would be an OK solution as well)

[dimbleby]

I doubt that poetry will ever lock a single package split across two sources

separate indexes are used for wheels and sdists

I have never heard of anyone doing this. If you are doing this: advise don't!

[f3flight]

Thank you for the advice =) This is just my state of things right now, I cannot change this without much effort. I personally don't see any issues with locking "across sources", my ask here is to make poetry attempt to find both sdist and wheel for locking purposes, instead of picking files only from the first satisfactory source. Or provide a way to lock versions only, w/o urls/sources - similar to "requirements.txt".

[dimbleby]

I understand the ask, I am telling you that I don't expect it ever to happen.

[MichaIng]

Just adding another concrete use case:

• On ARMv6hf and ARMv7 systems, most wheels need to be compiled, since upstream does usually not upload wheels for this architectures, but at best ARMv8/aarch64.
• Naturally this architecture is usually used by small slow SBCs, on which compiling wheels with C++ and especially Rust bindings, takes extra long, can fail due to insufficient memory etc. Even just installing the build dependencies takes long and can take significant consume resources on those boards already.
• When running Debian (or Raspbian/Raspberry Pi OS), luckily there is https://piwheels.org/, which aims to provide compiled wheels for all packages available on PyPI. A massive time and resource saver!
• Trivial to use with pip via extra-index-url, in which case it uses piwheels for better matching bianry wheels, and PyPI otherwise.
• Poetry however cannot use piwheels properly:
  • When adding it as "supplemental" index, it is never used, since generally every package is available on PyPI. But Poetry also pulls and compiles sources then, even if binary wheels are available on piwheels for the exact same version of the package.
  • When adding it as "primary" index, it is always used, since every package is generally available on piwheels as well. But not all versions of all packages have successful builds/wheels on piwheels, so it is necessary that PyPI is used as fallback for this. Poetry does not seem to support this, but would use PyPI only if a package is not registered at all on piwheels, i.e. not with any version.

I guess this is also the reason why #4816 cannot work: for each package, only a single source is possible, not only for the generated poetry.lock, but in general (while the first surely implies the latter).

For those cases, it would be really great if Poetry could behave closer to pip, generally searching multiple indexes for each package (if declared with respective priority), preferring PyPI if no (better suitable) binary wheels is available elsewhere, but pulling binary wheels from other indexes, if PyPI provides the tarball only.