Multiple calls to `--platform` in pip download is inclusive not exclusive

[ion-elgreco]

Description

According to the help (docs) it says you can supply --platform multiple times but only the first one is actually used to resolve the package, take this line:

 pip download nvidia-cusparse-cu12==12.5.4.2 --no-deps  --platform "manylinux_2_17_x86_64" --platform "manylinux2014_aarch64" --python-version 3 --abi none --implementation py

This should actually fetch this package: nvidia_cusparse_cu12-12.5.4.2-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl

But it actually fetches this one:

File was already downloaded ./nvidia_cusparse_cu12-12.5.4.2-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl

Expected behavior

Use every --platform arg to resolve the package.

pip version

25.0.1

Python version

3.11

OS

MacOS Sequoia

How to Reproduce

pip download nvidia-cusparse-cu12==12.5.4.2 --no-deps --platform "manylinux_2_17_x86_64" --platform "manylinux2014_aarch64" --python-version 3 --abi none --implementation py

Output

No response

Code of Conduct

• [x] I agree to follow the PSF Code of Conduct.

[ichard26]

This is a known limitation of the download command. It has similar semantics to pip install, resolving dependencies and selecting only one version for each package. Specifying --platform simply expands the pool of distributions it can consider.

This is already noted in the documentation, but we should make it more obvious. https://github.com/pypa/pip/issues/13254

[ion-elgreco]

pip download --help gives this though:

  --platform <platform>       Only use wheels compatible with <platform>. Defaults to the platform of the running system. Use this option multiple times to specify multiple platforms
                              supported by the target interpreter.

It says it can be used multiple times which is unintuitive, I thought it meant only have those that match both

[ion-elgreco]

@ichard26 considering this example nvidia_cusparse_cu12-12.5.4.2-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl is it possible with pip to force this download?

[notatallshaw]

I thought it meant only have those that match both

My understanding is that candidates may match either, specifying multiple platforms allows each platform to be a candidate to resolve to, but doesn't require the candidate matches all platforms specified.

I'm not familiar with the use case, but I'm quite sure people are relying on this behavior and it can't be changed without breaking workflows.

considering this example nvidia_cusparse_cu12-12.5.4.2-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl is it possible with pip to force this download?

If there are also other wheels for "manylinux2014_aarch64" and "manylinux_2_17_aarch64", then I don't think the --platform flag can guarantee you will get this wheel, just that it will be included as a candidate.

[pfmoore]

I feel as though the best option might be to deprecate multiple use of --platform, so that users would just run a pip download command for each platform. The resolver isn't designed in a way that would let multiple platform options work the way most users would expect.

[notatallshaw]

My concern is either it was implemented the way it was because there's some important use case that's been forgotten, or even if it was implemented without a specific use case in mind that an important one might have developed without pip maintainers knowing.

So if it's depreciated I would argue for an extended period, e.g. 1 year, to allow someone to report such.

[ion-elgreco]

I thought it meant only have those that match both

My understanding is that candidates may match either, specifying multiple platforms allows each platform to be a candidate to resolve to, but doesn't require the candidate matches all platforms specified.

I'm not familiar with the use case, but I'm quite sure people are relying on this behavior and it can't be changed without breaking workflows.

considering this example nvidia_cusparse_cu12-12.5.4.2-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl is it possible with pip to force this download?

If there are also other wheels for "manylinux2014_aarch64" and "manylinux_2_17_aarch64", then I don't think the --platform flag can guarantee you will get this wheel, just that it will be included as a candidate.

I use pip download, to fetch all wheels for a specific version that are from a certain platform. I do that by parsing the pypi metadata and then fetch those wheels through a private index, but it seems the edge case of multiple platforms can't be covered

[pfmoore]

My concern is either it was implemented the way it was because there's some important use case that's been forgotten

My point is that if it currently doesn't work, I'd support deprecating and then removing it rather than trying to fix it (because a fix would be complex, if it's possible at all).

If there's an important use case that's been forgotten, that implies that it works, so my point doesn't apply. But this issue is explicitly saying that it doesn't work, so I don't see how anyone could be relying on it 😕

[notatallshaw]

If there's an important use case that's been forgotten, that implies that it works, so my point doesn't apply. But this issue is explicitly saying that it doesn't work

It works in the sense it does something, I just don't know if that something is helpful to anyone or not, or if it's what was intended when implemented.

[pfmoore]

Ah, OK. I interpreted the OP's description as "any use of --platform after the first is ignored" (and I was advocating deprecating extra uses, rather than silently ignoring them). Apologies for the noise.

[Czaki]

Is there an option to change --platform to match any compatible platform tag? For now, if I specify --platform macos_14_0_arm64 it tells me that there is no compatible package even if it has macos_10_14_universal2 wheel. Similar problems with manylinux.

I was thinking that I may prepare a list of compatible tags and pass them via multiple --platform flags, but it does not work.

So it means that pip download can not download dependencies of any more complex project (for example, PyQt6 has a universal wheel where its dependency PyQt6-Qt6 ships separate arm and intel wheels).

I'm looking for a simple solution that will allow to prefetch wheels for tutorials (at conferences, where there may be a problem with the internet). And for now it looks like it needs separate Linux and macOS machines with not too new operating systems.

As pip install is able to determine the current platform tag and download, based on it, wheels with different platform tags, it should not be a big deal?

If someone points me to where I should look, I may try to make such a contribution.

[jsirois]

One place you could look for now is Pex, it supports these use cases directly with almost 100% of the hard work done by Pip.

For the OP from @ion-elgreco, for example:

:; pex3 download nvidia-cusparse-cu12==12.5.4.2 --intransitive --platform manylinux_2_17_x86_64-py-313-any --platform manylinux2014_aarch64-py-313-any --dest-dir /tmp/example

:; ls -l /tmp/example/
total 422868
-rw-rw-r-- 1 jsirois jsirois 216451135 Jul 31 08:14 nvidia_cusparse_cu12-12.5.4.2-py3-none-manylinux2014_aarch64.whl
-rw-rw-r-- 1 jsirois jsirois 216561367 Jul 31 08:14 nvidia_cusparse_cu12-12.5.4.2-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl

The 2 differences from Pip are:

1. Instead of a --platform --python-version --abi --implementation quad of options, you pass one --platform option that has syntax containing those 4 components (See: https://docs.pex-tool.org/buildingpex.html#platform).
2. The --no-deps option is spelled --intransitive.

Otherwise it even uses Pip to do the actual downloading under the covers and does so via minimal runtime patches. It runs 2 Pip jobs in parallel, one for each --platform. Use --pip-version to select from amongst all Pip versions released since 2022 - including 25.2 which the just-released Pex 2.49.0 supports.

For more complex cases like @Czaki, instead of using --platform, which can only give an approximate view of the target system you are downloading for, you can use --complete-platform instead. This option takes a json file argument and you can generate that json file by running pex3 interpreter inspect --tags --markers once on each target machine (See: https://docs.pex-tool.org/buildingpex.html#complete-platform)

Re:

And for now it looks like it needs separate Linux and macOS machines with not too new operating systems.

You still need this, but just 1 time to create a complete platform json description of those old machines. You can then re-use those complete platform files going forward for pex3 {download,wheel} ... resolves, etc.