Respect the active virtual environment instead of the installed location

[fmigneault]

Describe the feature

If pipdeptree is installed in the system python location (to make it always accessible), it will by default use the system python to look for packages. When a virtual environment is activated, this causes a confusing situation where packages listed do not correspond to the activated environment.

To work around this, a user has to manually provide the desired python:

# activate env
conda activate <env>  # or other virtualenv equivalent
# look for package
pipdeptree -p <package> --local-only --python "$(which python)"

Installing pipdeptree as a system-wide utility seems relevant to avoid reinstalling it each time for each environment (similar to what poetry recommends).

It seems to me that pipdeptree should prioritize the "active" python rather than its installed location.

[kemzeb]

Thanks for the feature request!

If pipdeptree is installed in the system python location (to make it always accessible), it will by default use the system python to look for packages. When a virtual environment is activated, this causes a confusing situation where packages listed do not correspond to the activated environment. . . . It seems to me that pipdeptree should prioritize the "active" python rather than its installed location.

Assuming I understand you correctly, I'm not sure if it's possible to have the pipdeptree console script choose the interpreter to be used to run its code. When the system-wide pipdeptree is ran, it will always use the system-installed Python (or a venv python if it was installed in a virtual environment). What is looks like when running cat $(which pipdeptree):

#!/usr/local/bin/python
# -*- coding: utf-8 -*-
import re
import sys
from pipdeptree.__main__ import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

Notice the shebang line. This would mean we would need to instead programmatically detect that the user has activated a virtual environment. I did think of one approach while looking at the venv docs. When the virtual environment is activated, it would set the environment variable VIRTUAL_ENV to an absolute path to the environment:

$ source venv/bin/activate
(venv) $ echo $VIRTUAL_ENV
/workspaces/pipdeptree/venv

From here we could snatch the path to the environment's interpreter and use it to grab it's associated packages. The only problem with this is I'm not sure yet if the poetry, conda, or others who fire up environments will set $VIRTUAL_ENV. Open to other approaches.

[fmigneault]

What if the shebang was replaced by #!/usr/bin/env python (not sure how to do that with the installation script) ?

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import sys
import subprocess

if __name__ == '__main__':
    sys.exit(subprocess.Popen(['which', 'python']))

Calling this echos the python path of the activated environment or default to the system if none is activated.

[gaborbernat]

Not cross platform or supporting pypy.

[gaborbernat]

Also this would need to be an option flag or breaking change and major version bump.

[kemzeb]

What if the shebang was replaced by #!/usr/bin/env python (not sure how to do that with the installation script) ?

This sadly won't work as one problem I can see with this would be that the virtual environment will need to have pipdeptree, pip, and packaging for it to run successfully and there's no guarantee that it does.

[fmigneault]

I'm confused. I thought #!/usr/bin/env python was more portable since it lets env figure out which is the first compatible python in PATH regardless of where it is specifically installed on a given OS. I must admit, I'm not familiar with pypy's case though.

This sadly won't work as one problem I can see with this would be that the virtual environment will need to have pipdeptree, pip, and packaging for it to run successfully and there's no guarantee that it does.

Isn't that the current situation if --python is not specified?

[gaborbernat]

I'm confused. I thought #!/usr/bin/env python was more portable since it lets env figure out which is the first compatible python in PATH regardless of where it is specifically installed on a given OS. I must admit, I'm not familiar with pypy's case though.

This sadly won't work as one problem I can see with this would be that the virtual environment will need to have pipdeptree, pip, and packaging for it to run successfully and there's no guarantee that it does.

Isn't that the current situation if --python is not specified?

That only works on Unix not on Windows.

[kemzeb]

Isn't that the current situation if --python is not specified?

I think you may have meant if --python is specified. When you give us a path to an interpreter, all we do is fire up a child process that "queries" your interpreter for its paths so that we can snatch its package metadata (i.e. /your/python -c "import sys; print(sys.path)"). There is no need to have pipdeptree and its dependencies installed.

[fmigneault]

No, I meant is not.

The idea was that, if pipdeptree is installed in the system python, it would always use that reference and looks for system packages. When the user wants to instead list packages from a virtual environment without using --python explicitly each time, their alternative is to install pipdeptree directly in the environment. Doing so, pipdeptree, pip and packages should be installed there, otherwise they wouldn't achieve the intended result anyway.

When --python is included, the issue is the same if the resolved pipdeptree happens to be installed in the virtual env (ie: <env1-path>/bin/pipdeptree --python <other-env>/bin/python ...) also requires that pipdeptree, pip and packages are all installed in that env1 virtual environment.

[kemzeb]

I see, you are saying that without specifying the --python option users would have to have pipdeptree and its deps installed in the environment in which they are running in (whether that be the system or virtual env). Yes, this is the case and is the only other option besides specifying --python as it is mentioned in the README.

Currently, I do not see any other approach to this besides looking for environment variables, i.e. $VIRTUAL_ENV. I would need to investigate other projects (e.g. poetry, conda) that implement virtual environments and see if they have this env var or something similar.

If this works out, I think one way of avoiding a breaking change or adding an option flag is by passing something like:

pipdeptree --python auto-detect # or --python auto

Here we would run a routine trying to determine what virtual environment implementation they are using by searching for certain environment variables. If all else fails, warn the user that we couldn't detect any virtual environment and fallback to using the system python. We could also just consider this a failure and exit. As I mentioned, this approach depends on whether the other implementations do set certain env vars that map to paths we could use to get their interpreter. And another thing, if these implementations make changes to this in the future, we would need to account for this (possibly also consider different versions of the implementation).

[fmigneault]

I like the environment variable approach. Seems more reliable. For conda, it should be CONDA_PREFIX I believe.

I think --python auto is a good compromise for non-breaking changes. If there should be auto-detection of some env variable such as PIPDEPTREE_PYTHON=auto, that would be even better. It could allow someone to set this somewhere in their .bashrc (or equivalent) and have the desired behavior without adding the option each time.

[kemzeb]

After further thought I do think this is worthwhile to pursue. I don't expect these virtual environment implementations to frequently change the way that we can snatch the env's prefix as users actively rely on them (i.e. this shouldn't be a maintenance burden). I'll tackle this when I get the time.

[kemzeb]

You should be good to go after I release 2.21.0!

I'm kind of iffy about introducing an environment variable, mainly because --python auto doesn't fallback on the system environment if we can't find a virtual environment (so you would have to unset that env var if you want to output the global packages) . Feel free to open a new issue about it if you want to discuss it or if you encounter any bugs with the current implementation.

[fmigneault]

Works great. Thank you for the new feature! One thing that could be useful is to display the resolved environment when --python auto is used (or maybe even every time regardless?), just to validate the result is as expected. Something like a log message Resolved Python: [/abs/path/to/python] before the package tree listing would do the trick.

[kemzeb]

Yes it does make sense to print an info message in this case; I'll open up a PR for it when I can.