how should we handle environment upgrades

[Mariatta]

Not sure if this is the right place to ask.

I tried updating the python version in one of my virtualenv environment today. The environment venv had Python 2.7 originally, and I wanted to update it to Python 3.6

This is what I did:

$virtualenv -p /Library/Frameworks/Python.framework/Versions/3.6/bin/python3 venv

The output from this command is:

Running virtualenv with interpreter /Library/Frameworks/Python.framework/Versions/3.6/bin/python3
Using base prefix '/Library/Frameworks/Python.framework/Versions/3.6'
New python executable in /Users/Mariatta/venv_exp/venv/bin/python3
Not overwriting existing python script /Users/Mariatta/venv_exp/venv/bin/python (you must use /Users/Mariatta/venv_exp/venv/bin/python3)
Installing setuptools, pip, wheel...done.

I'm curious of what this message mean: Not overwriting existing python script /Users/Mariatta/venv_exp/venv/bin/python (you must use /Users/Mariatta/venv_exp/venv/bin/python3)

Sounds like a warning or an error?

But when I activated venv, I'm getting python 3.6 accordingly.

$ source venv/bin/activate
(venv) $ python
Python 3.6.0b4 (v3.6.0b4:18496abdb3d5, Nov 21 2016, 20:44:47) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 

Just curious if the message is intended? What does it really mean?

Thanks :)

Using virtualenv 15.0.3

[cprkrn]

Don't think it's a big issue. Same thing here because it's messing with my Anaconda installation.

[0cjs]

I had no idea that one could change the Python version in a virtualenv by re-running the build command. Is this documented somewhere?

Anyway, when virtualenv venv is activated, what do the following commands print?

python3 --version
python --version

It looks to me as if it may have left python as a Python 2 interpreter and added python3 as the new Python3 interpreter. That might allow you to have both versions available in a virtualenv, with names matching a typical native package system installation on many Linux distributions.

[Mariatta]

The -p is documented: https://virtualenv.pypa.io/en/stable/reference/#cmdoption-p

[0cjs]

I am quite familiar with -p; the documentation I was asking about was the ability to reinstall an existing virtualenv.

[pradyunsg]

I'm not familiar with virtualenv's internal workings but my understanding is that "updating" a venv to a newer version of Python would require virtualenv to do some other work (like making sure all the packages are compatible w/ the newer version). I'm not sure that it does all that.

FWIW, when I try to upgrade, I delete the existing virtualenv and make a new virtualenv (with the newer Python) in the same directory and re-setup my environment by installing the requisite packages. If you need to have access to multiple python versions, I'd suggest you use a tool that helps deal with multiple virtualenvs for a project, like nox or tox.

[0cjs]

I did a little checking, and clearly this "upgrade" idea is Something That Should Not Be Done. I created a Python 3.5 virtualenv and then re-ran virtualenv on the same directory with -p pointing to a Python 3.6. I got the same Not overwriting existing python script /home/cjs/zzz/v35/bin/python (you must use /home/cjs/zzz/v35/bin/python3) message, but upon looking in bin/ it was clear what's going on: bin/python is a symlink to python3. A new lib/python36 dir was created with just the regular initially installed stuff, but without pytest (which I had added before the upgrade). left behind was bin/pytest which still had a shebang for this venv (#!/home/cjs/zzz/v35/bin/python3) but of course that doesn't run because from pytest import main fails, that being installed under lib/python35, not the lib/python36 that the new interpreter binary is using.

Perhaps we want to have virtualenv do a bit of checking and refuse to install (unless forced) in a dir that looks as if it already has been installed by virtualenv?

[gaborbernat]

I would say we probably want to drop upgrade entirely. If different python target we should do a fresh recreate.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Just add a comment if you want to keep it open. Thank you for your contributions.

[0cjs]

I agree that we want to drop the idea of upgrading. I'm not sure a "fresh recreate" can even work without removing the entire virtualenv first, since it seems hard to ensure that nothing pointing to the old environment is left behind.

[gaborbernat]

I think here the deliverable is how should we handle upgrades; e.g. when we're running without clear against a folder that has some interpreter already there. We should probably, overwrite if the home of the source is the same, refuse/clear otherwise - out of this, for now, I'm leaning towards implicit clean, but display a warning about triggering an implicit clean.

[0cjs]

@gaborbernat By "source" you mean that the path from which the python binary comes is the same as it was previously? This actually makes sense to me, since if you've got virtualenvs installed from a development version of Python (/home/cjs/work/mypython/...) you may rebuild and reinstall that Python, in which case you'd want to be able to easily update the copies of its files in the virtualenv, too.

[gaborbernat]

Seems no longer an issue as no one reported it in 3 years.

[0cjs]

I disagree; just because nobody's reporting it here and it's (hopefully!) a little-used feature does not mean that it's not broken. This is especially true for a feature like this where the breakage may be hard to detect.

I suggest we either properly define what it means to try to "upgrade" an existing virtualenv (which is a rather difficult thing) or, better yet, take the easy road and remove the "upgrade" functionality. The whole point of virtualenvs is that they are stable and easily recreated; upgrades break the former and the latter removes the need for upgrades.