Cannot uninstall packages using pip in cloned venvs

[willfurnass]

Example below uses virtualenv-clone HEAD, virtualenvwrapper 3.4-2 (Ubuntu 13.10 package) and virtualenv 1.10.1-1:

will@banana ~ $ mkvirtualenv testvenv1
New python executable in testvenv1/bin/python
Installing Setuptools..............................................................................................................................................................................................................................done.
Installing Pip.....................................................................................................................................................................................................................................................................................................................................done.
virtualenvwrapper.user_scripts creating /home/will/.virtualenvs/testvenv1/bin/predeactivate
virtualenvwrapper.user_scripts creating /home/will/.virtualenvs/testvenv1/bin/postdeactivate
virtualenvwrapper.user_scripts creating /home/will/.virtualenvs/testvenv1/bin/preactivate
virtualenvwrapper.user_scripts creating /home/will/.virtualenvs/testvenv1/bin/postactivate
virtualenvwrapper.user_scripts creating /home/will/.virtualenvs/testvenv1/bin/get_env_details
(testvenv1)will@banana ~ $ pip install nose
Downloading/unpacking nose
  Downloading nose-1.3.0.tar.gz (404kB): 404kB downloaded
  Running setup.py egg_info for package nose

    no previously-included directories found matching 'doc/.build'
Installing collected packages: nose
  Running setup.py install for nose

    no previously-included directories found matching 'doc/.build'
    Installing nosetests script to /home/will/.virtualenvs/testvenv1/bin
    Installing nosetests-2.7 script to /home/will/.virtualenvs/testvenv1/bin
Successfully installed nose
Cleaning up...
(testvenv1)will@banana ~ $ deactivate 
will@banana ~ $ python dev/virtualenv-clone/clonevirtualenv.py .virtualenvs/testvenv1 .virtualenvs/testvenv2
will@banana ~ $ workon testvenv2
(testvenv2)will@banana ~ $ pip uninstall nose
Not uninstalling nose at /home/will/.virtualenvs/testvenv1/lib/python2.7/site-packages, outside environment /home/will/.virtualenvs/testvenv2

yet

(testvenv2)will@banana ~ $ python -c 'import nose; print nose.__file__'
/home/will/.virtualenvs/testvenv2/local/lib/python2.7/site-packages/nose/__init__.pyc

[edwardgeorge]

Sorry, I don't think I got/saw a notification email for this issue. I'll see if I can reproduce and dig into what's going on.

[edwardgeorge]

weirdly, I can't reproduce this.

➜  ~  virtualenv-clone .virtualenvs/test1 .virtualenvs/test2
➜  ~  workon test2
➜  ~  pip uninstall nose
Uninstalling nose:
  /Users/edwardgeorge/.virtualenvs/test2/bin/nosetests
  [...]
Proceed (y/n)? y
  Successfully uninstalled nose

this is using a standard pip installed virtualenv and virtualenvwrapper on OSX, maybe it's something to do with ubuntu's packaged versions, perhaps they have slight differences to them? it's not unheard of. I don't have an ubuntu machine to test on presently. what happens if you use a pip installed virtualenv and a pip installed virtualenvwrapper instead of the ubuntu packaged versions? do you still get the same issue?

[edwardgeorge]

virtualenvwrapper versions tested were ==4.1.1, ==4.2, and ==3.4

[berdario]

FWIW, I tried to reproduce it on ubuntu 13.10, system virtualenv version 1.11, and I was unable to reproduce as well

[willfurnass]

The problem for me is that shebang lines aren't updated in various scripts in the cloned virtualenv. Following on from my example above:

will@banana ~ $ fgrep -RI testenv1 ~/.virtualenvs/testenv2/

/home/will/.virtualenvs/testenv2/local/bin/pip:#!/home/will/.virtualenvs/testenv1/bin/python
/home/will/.virtualenvs/testenv2/local/bin/easy_install-2.7:#!/home/will/.virtualenvs/testenv1/bin/python
/home/will/.virtualenvs/testenv2/local/bin/easy_install:#!/home/will/.virtualenvs/testenv1/bin/python
/home/will/.virtualenvs/testenv2/local/bin/pip-2.7:#!/home/will/.virtualenvs/testenv1/bin/python
/home/will/.virtualenvs/testenv2/local/bin/activate.csh:setenv VIRTUAL_ENV "/home/will/.virtualenvs/testenv1"
/home/will/.virtualenvs/testenv2/local/bin/activate:VIRTUAL_ENV="/home/will/.virtualenvs/testenv1"
/home/will/.virtualenvs/testenv2/local/bin/nosetests:#!/home/will/.virtualenvs/testenv1/bin/python
/home/will/.virtualenvs/testenv2/local/bin/nosetests-2.7:#!/home/will/.virtualenvs/testenv1/bin/python
/home/will/.virtualenvs/testenv2/local/bin/activate.fish:set -gx VIRTUAL_ENV "/home/will/.virtualenvs/testenv1"

[berdario]

Uhm, here it might be worthwhile to raise an exception or output something to stderr

I think that having shebangs that point to a python in a symlinked directory might also be a problem if moving the virtualenv from the symlink path, but this is probably not what is going wrong in this case

[willfurnass]

Think the problem stems from there being symlinks from .virtualenvs/testenv1/local/{bin,include,lib} to the .virtualenvs/testenv1/{bin,include,lib} that aren't updated when testenv1 is cloned, and /home/will/.virtualenvs/testenv2/local/lib/python2.7/site-packages coming before /home/will/.virtualenvs/testenv2/lib/python2.7/site-packages in sys.path once testenv2 is active.

[wuurrd]

I had this same issue, resolved by deleting the symlinks in local, and pointing them to the destination virtualenv.

[pjw91]

This patch might fix this issue, though I'm not familiar w/ the usage of "local" directory in Linux(?).

--- a/clonevirtualenv.py    2014-10-29 22:15:44.083003865 +0800
+++ b/clonevirtualenv.py    2014-10-29 22:18:29.798996574 +0800
@@ -7,6 +7,7 @@
 import subprocess
 import sys

+from virtualenv import fix_local_scheme
 version_info = (0, 2, 4)
 __version__ = '.'.join(map(str, version_info))

@@ -75,6 +76,8 @@
             (src_dir, dst_dir))
     shutil.copytree(src_dir, dst_dir, symlinks=True,
             ignore=shutil.ignore_patterns('*.pyc'))
+    shutil.rmtree(os.path.join(dst_dir, 'local'))  # NOTE: dirty hack, works on Ubuntu Precise for me
+    fix_local_scheme(dst_dir)
     version, sys_path = _virtualenv_sys(dst_dir)
     logger.info('fixing scripts in bin...')
     fixup_scripts(src_dir, dst_dir, version)

[edwardgeorge]

thanks. is that rmtree safe? could we be deleting things that may not be replaced by the fix_local_scheme call?

would it be safer to iterate through the symlinks in local and simply update the target location? there is symlink rewriting code in clonevirtualenv already.

[pjw91]

You're right.

In fix_local_scheme call, it'll check the existence of the local directory, if exists, no-op, otherwise, create it.

Thus, the best/correct solution is copy-modifiy-paste the code from fix_local_scheme to here, but I'm not familiar to the role of the local dir in linux. So I put the code here instead of a PR.

[jmuhlich]

Note that package upgrade is also broken, not just uninstall.

[pjw91]

The symlink problem seems to be fixed by #17