Deprecate pip, pipX, and pipX.Y

[dstufft]

Currently, people are regularly running into problems over confusion about which particular Python a particular invocation of pip is going to manage. There are many cases where what Python a particular invocation of pip* is going to invoke is unclear:

• pip3 install --upgrade pip will overwrite pip and possibly switch it from pointing to 2.7 to 3.5.
• pip3 isn't specific enough, you might have 3.4 and 3.5 installed.
• pip3.4 isn't specific enough, you might have 3.4.0 and 3.4.1 installed.
• pip3.4.0 isn't specific enough, you might have multiple copies of 3.4.0 installed in various locations.
• We don't have a good answer for what the pip binary should be called on alternative Python interpreters like PyPy (pip-pypy? What if we have two versions of PyPy? pip-pypy2.6? What if we have PyPy and PyPy3? pip-pypy-2.6 and pip-pypy3-2.6?).

Overall, it's become increasingly clear to me that this is super confusing to people. Python has a built in mechanism for executing a module via python -m. I think we should switch to using this as our preferred method of invocation. This should completely eliminate the confusion that is caused when python and pip don't point to the same version of Python, as well as solve the problem of what do you call the binary on alternative implementations.

In addition to the confusion, we also have the fact that pip install --upgrade pip doesn't actually work on Windows because of problems with the .exe file being open. However python -m pip install --upgrade pip does work there.

I see only three real downsides:

• There is a lot of documentation, examples, code or other instances of inertia using just pip and this will be churn for those.
• It's 10 more letters to type.
• It doesn't work on Python 2.6.

For the first of these, I think the answer is to just have a very long deprecation cycle, in the order of years. I wouldn't even put a specific date on it's removal, I'd just add the warnings and re-evaluate in the future. Luckily we've shipped support for python -m pip for quite some time, so it won't be something that people need to deal with version differences (mostly).

The second of these I don't really have a great answer for, I think that 10 extra letters probably isn't that big of a cost to pay for the reduced confusion and the default answer working on Windows. We could possibly offer a recipe in the docs to restore pip, pipX, and pipX.Y via shell aliases.

This last item is the biggest sticking point for me. As far as I know, Python 2.6 still has far too many users for us to drop it since, as of 6 months ago, it was still ~10% of the traffic on PyPI (source). The problem with Python 2.6 is that it only supports -m when the target is a module not a package. I see four possible solutions to this:

• Don't deprecate pip* on Python 2.6.
• Add a module like pipcli.py that people can invoke like python -m pipcli instead of python -m pip.
• Move pip/ to _pip/ and make pip.py.
• Document that due to limitations of Python 2.6, it will need to be invoked as python -m pip.__main__.

I don't really like the pipcli idea, the other three all have pros and cons, but I think I personally live with either not deprecating pip* on Python 2.6 and/or documenting that it needs to be invoked as ``python -m pip.main on Python 2.6).

What do the @pypa/pip-developers think?

[RonnyPfannschmidt]

i like the move from pip to _pip, that way pip's implementation goes to a more "private" namespace the expense is breaking tools that go into pips internals

[dstufft]

The other expense is that if we ever want to make a public API we're either limited to having a single namespace (whatever is in pip.py) or we need to change it back to a package (and possibly break Python 2.6 again, unless we've deprecated it by then).

Of course, we may never make a public API in which case, the point is moot.

[dstufft]

I can't help but think that @warsaw and @ncoghlan probably have some opinions on this too.

[dstufft]

Maybe @bkabrda too! and @tdsmith

[Lukasa]

Don't underestimate the power of that first point. The intertia is high: lots of tools will assume pip, and lots of documentation will be wrong. Having a long deprecation cycle is basically mandatory here. Otherwise, I think this is a good idea: +1.

[dholth]

-1 on removing pip I would have to change all of my deployment scripts. +1 on removing pipX and pipX.Y

On Tue, Oct 6, 2015, at 08:16 AM, Cory Benfield wrote:

Don't underestimate the power of that first point. The intertia is high: lots of tools will assume pip, and lots of documentation will be wrong. Having a long deprecation cycle is basically mandatory here. Otherwise, I think this is a good idea: +1.

— Reply to this email directly or view it on GitHub[1].

Links:

1. https://github.com/pypa/pip/issues/3164#issuecomment-145839321

[dstufft]

Don't underestimate the power of that first point. The intertia is high: lots of tools will assume pip, and lots of documentation will be wrong. Having a long deprecation cycle is basically mandatory here.

Yea, completely agree. I essentially assume that we should not have a defined removal date (and possibly never) and just have it log a message to stderr.

[Lukasa]

I essentially assume that we should not have a defined removal date (and possibly never) and just have it log a message to stderr.

I don't think that will work. Just printing annoying warnings with no defined "your shit will break no later than X" date doesn't really help: people will just ignore the warnings. I think if you want to do this you should decide a date (possibly one far away, but still). One possible date to start the discussion: when the last RHEL LTS release with Python 2.6 stops being supported (that's very far away still, but worth discussing).

[dstufft]

-1 on removing pip I would have to change all of my deployment scripts. +1 on removing pipX and pipX.Y

I don't think it makes sense to deprecate (not talking about removal any time soon) pipX and pipX.Y without also doing it for pip since that is arguably the worse offender of them all.

[piotr-dobrogost]

One possible date to start the discussion: when the last RHEL LTS release with Python 2.6 stops being supported (that's very far away still, but worth discussing).

This is good idea.

[dstufft]

One possible date to start the discussion: when the last RHEL LTS release with Python 2.6 stops being supported (that's very far away still, but worth discussing).

That's November 30, 2020 for the end of RHEL 6 Production 3 phase. They have a super special extended life cycle beyond that, but perhaps we could just target 2020 and if we roll around to 2020 and Python 2.6 is still somehow in wide support, we push it back further.

[dholth]

Another alternative would be to stop requiring that pip execute in the same Python environment that it is installing things into...

[dstufft]

To be honest, I'm not sure how pip install -p python2.7 is any better than python2.7 -m pip install. We have to inspect the Python we're installing into to get information from it, so either we're going to subshell into that Python to shuffle data back and forth (like my half done virtualenv rewrite does) or we'll need to continue to be executed by the same Python environment. Feels like shuffling deck chairs more than anything else.

[dholth]

For that particular idea the main benefit would be that you would only have to upgrade pip once.

[dstufft]

That much is true, the flip side of that is it makes it (somewhat) harder to support versions of pip older than what pip itself supports, since the installs are no longer independent (or you'll need to keep around an older copy installed somewhere else). On the other hand, pip could more easily drop support for running the main pip binary command in a particular Python, while keeping compatibility for installing into that version of Python. It would (continue) to enable bundling all of pip into a single zip file that can be executed somewhat independently of actually having it installed.

It doesn't address the fact that pip install --upgrade pip on Windows blows up because the OS has a handle open to pip.exe though, which I don't think can be solved without using python -m, at least if I understand @pfmoore correctly.

[bkabrda]

I think @sYnfo wants to comment on this more than I do, since I no longer maintain Python in Fedora/RHEL.

[dstufft]

Ah, that's right, I forgot. Sorry!

[bkabrda]

(But I personally think that even if these are removed, we'll still provide them on distribution level in form that is best for the set of Python interpreters that we ship; at least that was my first idea when I read the proposal... We have a general policy for Python executables that mandates this in Fedora.)

[pfmoore]

• +1 for deprecation - but I'd be happy enough simply to deprecate in documentation, not make the scripts themselves moan at the user.
• It's only 6 more letters on Windows - py -m pip :-)
• For 2.6, I'm OK with either letting people continue to use pip or advising python2.6 -m pip.__main__. I don't think it's worth making changes to pip to give them any other solution.

The inertia issue is huge, and I don't think we should fight it directly. Rather, we should switch the documentation to use the "python -m pip" form, and make that form official PyPA policy (by which I mean we take pains to use that form consistently in whatever we post, etc). Maybe offer PRs for the install documentation of well-known projects to switch them to the new form. We can worry about formally deprecating and/or removing the scripts once the python -m pip form starts to gain a bit of traction in common usage.

[tdsmith]

I think my success criterion is making this the incantation that people get from StackOverflow. Shoot for the moon, etc etc.

I think publishing a linkable intent-to-deprecate message with a rationale may help convince third-party maintainers to accept documentation PRs.

The messaging here is tricky, because deprecating pip doesn't mean deprecating pip...

[dstufft]

I think the only real alternative is Daniel's suggestion. I think the current situation sucks and I can't really think of a way to save the attempt to manage pip versions using a suffix or prefix or anything that doesn't end up actually specifying which interpreter you want to run under.

[sYnfo]

I'm +1 for deprecation—it seems to make a whole lot of sense from upstream point of view and I don't see any issue this could cause downstream.

In Fedora this would mean shipping all the binaries during the deprecation period, as we do now; and not shipping them at all afterward. Perfect sync with upstream, hopefully. :) I'll make sure all the Fedora docs get updated, when this goes official.

( @bkabrda If I understand the guidelines correctly, they only mandate shipping all the MAJOR.MINOR executables iff there are any executables in the first place, this issue seems to be about removing the pip executables entirely, right? )

Also +1 to either keeping pip or using python -m pip.main with python 2.6.

[erikrose]

A couple of points that haven't come up:

• Doesn't this just move the problem to python? Why is python (of various versions and in various virtualenvs) any easier to keep track of than pip?
• If you're using a virtualenv (which most people should be), pip unambiguously refers to the active one. An alternative solution would be to encourage people to always use venvs, which solves other problems as well, e.g. dependency conflicts from dropping everything into a global environment.

[rschoon]

I'm not big on the idea, but I don't think I've ever run pip outside of virtualenv, which is usually activated into my current shell, so the extra typing doesn't gain me anything (I'm selfish!). Of course, if it's not activated, I probably have to type a lot more than that anyway...

How would people feel about making an exception for virtualenvs? There's no mystery for which python it is using, but then again it might be too confusing to have it work differently between for virtualenv and the system install, so I'm feeling pretty humble about its quality as a suggestion. I'm also not sure there's a good solution for checking that.

Also,

pip3.4 isn't specific enough, you might have 3.4.0 and 3.4.1 installed.

Does this really come up often? I'm curious how you're distinguishing between them if so, since I didn't think python would normally install itself any more specific than by the pythonX.Y name (and unless they have different prefixes, sounds like it would also try to share site-packages anyway).

[rschoon]

Oh, I guess that does apply having multiple 3.4.0 installs too, but then it seems like you'd definitely be using full paths to distinguish whether you're using "pip" or "python -m pip", but since you're using full paths anyway, the argument against having extra wordiness goes away.

[mattrobenolt]

I may be a little disconnected here, but why not replace pip with an alias that was effectively something like:

alias pip=/usr/bin/env python -m pip

Or a script that was akin to this to be installed into /usr/local/bin/pip.

That way we don't lose pip, and it universally works errywhere.

Also not sure if that solves the problems you're facing, just my $0.02. :rage3:

[dstufft]

@erikrose It does sort of shift the problem to python, but typically the confusion that I've seen arise stems from when python and pip disagree. Like you might have /usr/bin/python and /usr/local/bin/python and if python points to /usr/local/bin/python and pip points to /usr/bin/python it's a recipe for confusion. So we completely eliminate the confusion cased by having the two disagree, but removing any ability for it to disagree. You're still left with confusion about what python means but I don't think there's anything at all that can be done about that, and particularly not by us.

A virtual environment makes the issue harder to hit, but I think that it's still possible. My gut tells me that new users (the ones most likely to get tripped up by this) are probably not going to be religiously using virtual environments (if they're using them at all).

@rschoon We could possibly do that, it'd require a permanent special case inside of pip when pip installs itself (because there's no setup.py in a virtual environment) but it's certainly a possibility. My main fear with that is it feels like it'd just always be better to use python -m pip anyways because it works inside and outside of a virtual environment instead of having to remember to switch commands based on whether you're in one or not.

I'm not sure exactly how often the 3.4.0 and 3.4.3 thing comes up. I know that it's not super unusual on OSX since you have system provided Python sitting in /usr/bin/python and you might also install the Python.org installer or Homebrew or Macports or pyenv (or some combination of the above).

[dstufft]

@mattrobenolt

Basically, because it's super confusing if you're doing something like running myvenv/bin/pip without activating the virtual environment first or if you have a copy of Python installed to a non standard location /opt/my-app and you run /opt/my-app/bin/pip and expect to manage /opt/my-app/bin/python.

[qwcode]

it feels like a loss for virtual environments to lose a basic pip script when they don't suffer from this problem.

as for real pythons I guess I wouldn't mind seeing a more exact pip-<python binary path> type console script, and let the distro packagers tack on simpler scripts for system-managed pips.

[Ivoz]

-1 on deprecating pip, +1 on the others. I want to expand on some framing thoughts I have, that might help for discussing what is such a large issue. Skip to the break if you just want to read my thoughts on solutions.

For a start, you want to look at why we have this problem, and is it analogous to anyone else? It comes from having and allowing multiple pythons on the one running system. So system package managers do not have this problem, because for instance, you don't (can't) run debian jessie and debian wheezy live at the same time on one system; so it doesn't need to manage a libreoffice3.5 and libreoffice4.3 for example. However many other language package managers start having to deal with the same problem as pip's when they too have multiple versions installed. As @erikrose mentions, even python itself already runs into this issue on deciding on what python now is when more than one is installed.

I also want to look at the issue from the POV of majority python users. Note that this has become a real pain point mostly (or more-so) for people with more than 2 pythons installed. Otherwise pip would work, or simply pip2 and pip3 (and there probably wouldn't be enough inertia for everyone to start discussing). Beyond that it starts getting really complicated. But I'd believe most python users are happy using just one python. Even if their system somehow gives them 2 at some point, or they manage to install multiple, instead of upgrading / (removing previous and installing a new one) - if they got things right, they'd be fine with only one python going, and by extension, that python's pip. For all of these users suddenly taking away pip makes absolutely no sense and is just painful.

The other big source of pain is when someone merely tries to install a new python over a previous existing one, but the story for the entire environment being migrated to the new one (or "taking over the old one") isn't there. In that case I want to make the distinction that this is the install story's fault, not the existence of multiple pythons. For instance someone hoping to "install a new python!" but not getting it on their path. Even uninstalling the old python, might do nothing about giving them the environment they want (the new python on their path).

Also note that while the number of users collecting problems with managing pips may be small, their complaints are the only ones heard. I'd venture that their opinions are probably the majority on this discussion as well (because they're the ones with the issue). The silent majority doesn't care until things get changed for them, but we should still look to represent their use case fairly. Not of course, that those complaints therefore can't be valid.

---

Now with that in mind, here's solutions I like:

• clearly pip<versionstuff> doesn't scale well at all as a solution. So I'm in agreement for removing it. Most of the time its better to wait for a decent solution (if it exists) than try and keep going with one that creates as many issues as it solves.

• Stop replacing pip (or other pip2s or pip3s, even) without asking. Even system package managers do this. They ask beforehand. So should we. This way at the very least, the user sees straight away that there's an issue and perhaps can make a decision for themselves what they want pip to be. One can do a lot with this - look at who's python installation the existing pip comes from - its the same one as the current pip trying to install itself, then this could be fine. Otherwise make the user say --yes or --replace or answer Y to a prompt. Note that this could help with the same problem with other userland-programs-that-come-from-pypi. Make sure the user wants the executable script replaced. Even if this means we have to wait a long time to tell people that they might have to interact after calling pip install by default (for scripted uses of pip), and give them time to add --replace-scripts (or w/e) to their callouts, so be it.

• Start outputting some information about where things come from in an install! This could solve a lot of issues straight away. If I install a package with pip, pip doesn't speak about

  • what python it's on/from
  • what version it is, where its installed
  • where it is installing the new package(s)

  This would all be super useful information to know. It will immediately show me if the pip I'm calling on the command line is not the actual pip that I want, which is what tricks a LOT of users. It will also show me I'm installing for the right python and into the right site-packages.

I especially believe implementing the last two points above, would remove a lot of average user-problems in relation to this issue. In many cases they would be empowered to know the problem and solution themselves.