Various package-index filtering flags do not affect the environment markers

[pradyunsg]

This is a common pattern that I'm seeing in pip's issue tracker; so filing an issue to (a) trigger a discussion of how to improve documentation / output / errors to better deal with the mismatch in user expectations vs behaviour here and (b) consolidate those issues.

Basically, the fundamental problem here is that pip has various options that affect the wheels that pip will consider when triaging things (--platform <platform>, --python-version <python_version>, --implementation <implementation>, --abi <abi>) but those do not affect the marker evaluation environment for dependencies.

This results in a subtle failure mode: pip install [options] package will use a wheel for package that has a (hypothetical) Python version 9.22 but when evaluating environment markers (https://packaging.python.org/en/latest/specifications/dependency-specifiers/#environment-markers), that would still use the environment for the current Python interpreter.

It is not possible to compute the environment markers based on the values passed in by the user via the CLI.

[pradyunsg]

https://github.com/pypa/pip/issues/10050#issuecomment-1113147833 has the relevant piece from that issue.

[pradyunsg]

I think the right move here is doing what pex has done, and taking a complete environment specification including environment markers; and not inferring anything from the interpreter when that's done. It'll be a slightly disruptive change, like any other change, but we'll be in a better spot in terms of this.

Thoughts on this?

[pfmoore]

Agreed, this sounds better. In fact, I'd like to go even further and completely isolate environment discovery - not just marker values, but sysconfig locations, and anything else that we introspect from the Python interpreter. That would allow us to manage any sort of environment, even "artificial" ones constructed from information supplied independently of the running interpreter.

I'm thinking of this in support of --python, or --target/"install scheme" types of option.

I'm happy to limit the change to markers as an incremental step, if that's easier to achieve, but I'd like to have our long-term goal be to have full isolation of "interpreter introspection".

[pradyunsg]

I really don't want to allow the user to specify arbitrary paths with schemes (this was extensively discussed during installer's API/CLI design discussions) because I'm wary of what hacky layouts people might expect to be "supported" if we allow that.

I am OK with exposing basically everything else, since misuse/abuse of them seems less likely/problematic to me.

[pfmoore]

I wasn't suggesting that paths get exposed directly as user options. Simply that internally, we collect everything that is introspected from the interpreter in one place. Then, flags for things like --platform can modify that internal structure. There won't be flags to alter the paths, but existing means of configuring paths, like --target or --user, can be modified to simply change the "interpreter config" structure, rather than having effects across the code base. It's not a big deal, though.

I absolutely agree, we don't want user options to individually specify paths. That way lies chaos...

[pradyunsg]

Oh, that wasn't clear. 😅

We're in complete agreement here. :)

[sbidoul]

I'm copying @uranusjr's UX proposal from https://github.com/pypa/pip/issues/9981#issuecomment-843046920 here:

pip install --marker="os_name='nt'" --marker="platform_release='...'"

I assume that whenever one --marker is provided, none of the other markers usually provided by the platform will be made available to the resolver. I suppose it would also make sense to rely only on explicit markers whenever any of --platform, --implementation or --python-version is used. Installation or build would fail if any specifier uses a marker that is not explicitly provided by the user.

[ichard26]

If it's feasible (I haven't checked), it may be worth it to emit a warning when a marker is encountered when a package index filtering option is given.

[jsirois]

It is not possible to compute the environment markers based on the values passed in by the user via the CLI.

@pradyunsg although it's true you can't compute them all, you can compute the ones that seem to be used the large majority of the time in the wild.

Pex has had (has) its own legacy --platform tag, and it's a string with it's own format that includes the 4 components of Pip's --platform, --python-version, --implementation, --abi. This (abbreviated) --platform idea was misguided and in early 2022 I added --complete-platform which is a json blob containing the full environment marker dict and full compatibility tag list for the foreign platform to fully characterize it for resolution purposes. That said, despite --platform being misguided, Pex tries to never break backward compatibility; so I came up with this to continue to support the abbreviated --platform, but better than Pip does natively with the 4 options: https://github.com/pex-tool/pex/blob/27c2db2bf26039bef41323c964bc4e0317a7b4f5/pex/pep_508.py#L18-L130

That derives as many environment markers as it can from the 4 platform values and leaves the rest None. Since most environment markers in the wild are python_version, this tends to work. When it does not there is other code that handles the None environment value accesses and provides a useful error message; something like:

Failed to resolve for abbreviated platform cp311-cp311-macosx_10_9_x86_64.
Resolve requires evaluation of unknown environment marker: 'platform_release' does not exist in evaluation environment.

Here the user passed --platform macosx_10.9_x86_64-cp-311-cp311 to Pex, which translates into --platform macosx_10_9_x86_64 --python-version 311 --implementation cp --abi cp311 for Pip.

All this is done using minimal runtime patches to off the shelf Pips from 20.3.4 to 24.1.1.

To underscore though, pex --platform ... / pip download --platform ... --python-version ... --implementation ... --abi ... were not well thought out and something like --complete-platform is the way to go from experience. Many users have shipped AWS and Google cloud lambda function zips from their Macs using --complete-platform for a few years now. If Pip can deprecate and drop those 4 options eventually, that seems sane.

[hackowitz-af]

This is a relevant issue for my team. We are having an incredible amount of difficulty using pip on an internet-connected computer to download packages to transfer to an air-gapped network.

Two notable issues:

• Specifying --platform does not affect any of the sys_platform markers.
• Specifying --only-binary=:all: fails to download sdist packages that would have worked on the target system

As someone who isn't too familiar with the internal workings and limitations of pip, I'd like specifying --platform, --python-version, etc. to simply download the packages for the specified environment.

[notatallshaw]

@hackowitz-af Yes, this is a known limitation of pip, and a non-trivial feature to implement, as pip is a volunteer project this feature is waiting on someone to propose and then implement a design that maintainers are generally happy with, until that happens this feature request will stay open.

However on your second issue:

Specifying --only-binary=:all: fails to download sdist packages that would have worked on the target system

I think you have a misunderstanding here --only-binary=:all: means that no sdists should be considered, so this is working as intended. If there is some feature you think is missing or some bug with this please open a new issue.

[jsirois]

@hackowitz-af If you're not using Windows, you might be able to use Pex for this. It uses Pip under the covers, but patches it to implement this feature as noted in https://github.com/pypa/pip/issues/10050.

[hackowitz-af]

@notatallshaw

--only-binary=:all: means that no sdists should be considered

Good catch, the fact is I don't actually want only binaries, but the current usage requires it:

ERROR: When restricting platform and interpreter constraints using --python-version, --platform, --abi, or --implementation, either --no-deps must be set, or --only-binary=:all: must be set and --no-binary must not be set (or must be set to :none:).

All I actually want when I use --python-version, --platform, --abi, or --implementation is to download packages to use in that environment. This seems different from the current definition, so I'll just make a new issue

[hackowitz-af]

@jsirois

Pex [...] uses Pip under the covers, but patches it to implement this feature.

Cool! I tried exploring this but didn't get very far. It seems pex is not really a pip stand-in, from the perspective of a first-time user. Is there an example analogue to pip download -r requirements.txt --platform=<some-other-platform> ... that gives the same result as pip except setting and using the environment markers as expected?

[jsirois]

@hackowitz-af I've started https://github.com/pex-tool/pex/discussions/2789 for this discussion re: possible Pex use.

[allanlewis]

@hackowitz-af if your requirements.txt is comprehensive, you can just add --no-deps; otherwise you could use pip-tools to generate a comprehensive lock-file and then download or install dependencies from that with --no-deps.

[jsirois]

FWIW, re: https://github.com/pypa/pip/issues/11664#issuecomment-3002029823, Pex 2.41.0 now has 1st class support for this via pex3 download where you can use --platform or --complete-platform to download distributions needed by foreign systems.