fix: remove imports from the deprecated pip API

[Midnighter]

• Fix #105

I think this should be a good start to move away entirely from pip. The only part I'm not clear about is the local_only and user_only flag. I'm not sure how to influence that for the pkg_resources.working_set.

[naiquevin]

Thanks for this. I'm away from work for a few days and have to take a proper look at this problem as well. Will update sometime next week.

[techalchemy]

@Midnighter thanks for taking a stab at this, I can shed some light on how this is handled (I would love to see this land as we are using pipdeptree in pipenv and I'm not a huge fan of patching things all the time). Note that there is a full example at the bottom but I haven't tested it at all, feel free to use it -- it should be an exact implementation of the logic underlying pip's own implementation

Firstly you have a few options, you can rely on pip-shims which I just added FrozenRequirement objects to specifically for this library in case that is the desired path, which is minimal in terms of changes right, but ultimately sticking with the pip internal API is not a sustainable path long term.

In the get_installed_distributions api (and any interface to pkg_resources.WorkingSet objects), you'll be dealing with either pkg_resources.DistInfoDistribution objects or pkg_resources.EggInfoDistribution objects:

import pkg_resources
dists = list(pkg_resources.working_set)
>>> print("[{dist.__class__!s}] : {dist!r}".format(dist=ws[0]))
[<class 'pkg_resources.DistInfoDistribution'>] : zope.deprecation 4.3.0 (/home/hawk/.pyenv/versions/3.7.0/lib/python3.7/site-packages)
>>> print("[{dist.__class__!s}] : {dist!r}".format(dist=ws[-1]))
[<class 'pkg_resources.EggInfoDistribution'>] : pythonfinder 1.1.0 (/home/hawk/git/pythonfinder/src)

The local_only argument does two things:

• Check if you are in a virtualenv by comparing sys.prefix to either the real_prefix (if set) or the base_prefix
  • If not, all visible packages are allowed and the logic short circuits
• If you are in a virtualenv, only return results that begin with the current interpreter's prefix (by normalizing both the distribution's path and the interpreter's path)

To see if you are in a virtualenv, it's just:

def running_under_virtualenv():
    real_prefix = gettattr(sys, "real_prefix", None)
    return True if real_prefix is not None else (sys.prefix != sys.base_prefix)

This gets a bit tricky because Egg distributions don't actually work that way, so you need to expand your search path when looking for those specifically. Roughly speaking, you need to have the rules:

• If in a virtualenv, add the normal site_packages directory from the python installation, then (if available) add the user site directory, and search those for egg-link files representing the distribution
• If not, invert the order and search
• Finally just use the dist.location as your comparison point

For simplicity this is the relevant code (roughly, not in any way tested):

import site
import sys

from distutils import sysconfig as distutils_sysconfig

site_packages = distutils_sysconfig.get_python_lib()

try:
    user_site = site.getusersitepackages()
except AttributeError:
    user_site = site.USER_SITE


def find_egg(egg_dist):
    search_locations = []
    search_filename = "{0}.egg-link".format(egg_dist.project_name)
    if running_under_virtualenv():
        search_locations.append(site_packages)
        if user_site:
            search_locations.append(user_site)
    else:
        search_locations.append(site_packages)
        if user_site:
            search_locations.append(user_site)
        search_locations.append(site_packages)
    for site_directory in search_locations:
        egg = os.path.join(site_directory, search_filename)
        if os.path.isfile(egg):
            return egg


def locate_dist(dist):
    location = find_egg(dist)
    if not location:
        return dist.location

That gives us a bunch of infrastructure to just add the following as our test function for local installations:

def is_in_environment(dist):
    if not running_under_virtualenv():
        return True
    return normalize(locate_dist(dist)).startswith(normalize(sys.prefix))

The user_only conditional does the same exact thing, except it checks whether the location starts with the normalized user_site path instead. So we can add one more test function:

def is_in_usersite(dist):
    return normalize(locate_dist(dist)).startswith(normalize(user_site))

And we can add a final summarizing function as there was before:

def dummy_test(dist):
    return True

def get_installed_distributions(user_only=False, local_only=True):
    test_local = is_in_environment if local_only else dummy_test
    test_user_site = is_in_usersite if user_only else dummy_test
    return [
        dist for dist in pkg_resources.working_set
        if test_local(dist) and test_user_site(dist)
    ]

Putting that all together:

import pkg_resources
import os
import site
import sys

from distutils import sysconfig as distutils_sysconfig

site_packages = distutils_sysconfig.get_python_lib()

try:
    user_site = site.getusersitepackages()
except AttributeError:
    user_site = site.USER_SITE


def running_under_virtualenv():
    real_prefix = gettattr(sys, "real_prefix", None)
    return True if real_prefix is not None else (sys.prefix != sys.base_prefix)


def normalize(path):
    return os.path.normcase(os.path.abspath(os.path.expanduser(path)))


def find_egg(egg_dist):
    search_locations = []
    search_filename = "{0}.egg-link".format(egg_dist.project_name)
    if running_under_virtualenv():
        search_locations.append(site_packages)
        if user_site:
            search_locations.append(user_site)
    else:
        search_locations.append(site_packages)
        if user_site:
            search_locations.append(user_site)
        search_locations.append(site_packages)
    for site_directory in search_locations:
        egg = os.path.join(site_directory, search_filename)
        if os.path.isfile(egg):
            return egg


def locate_dist(dist):
    location = find_egg(dist)
    if not location:
        return dist.location


def is_in_environment(dist):
    if not running_under_virtualenv():
        return True
    return normalize(locate_dist(dist)).startswith(normalize(sys.prefix))


def is_in_usersite(dist):
    return normalize(locate_dist(dist)).startswith(normalize(user_site))


def dummy_test(dist):
    return True


def get_installed_distributions(user_only=False, local_only=True):
    test_local = is_in_environment if local_only else dummy_test
    test_user_site = is_in_usersite if user_only else dummy_test
    return [
        dist for dist in pkg_resources.working_set
        if test_local(dist) and test_user_site(dist)
    ]

[naiquevin]

For now I have quick-fixed the ImportError by updating the import statement as per the changes pip version 18.1.

I would be interested in a long term fix such as this one. But not being a pipenv user I need to catch up with the developments happening there. Will have a look at pip-shims, I quite liked the idea upon a quick glance at it's README.

[techalchemy]

@naiquevin the fix outlined above is kind of independent of pipenv, it mostly relates to the fact that pip is maintaining an internal API that they are likely to continue breaking going forward. I already have fixes for all of the broken things merged into pipenv's master branch -- this was the change we used for the next release:

https://github.com/pypa/pipenv/blob/master/tasks/vendoring/patches/vendor/pipdeptree-updated-pip18.patch

[naiquevin]

Interesting approach. How do you make sure a shim is added when internal api changes? Or is it something that can only be done after some one reports a problem when a new version of pip is released?

[techalchemy]

https://travis-ci.com/sarugaku/pip-shims/builds/87356090 => it's now on a nightly cron to build against the master branch of pip, so ideally I'd have some warning when things break

[stonebig]

hi, with pip-21.3 coming in 2 weeks , get_installed_distributions if fully removed : https://github.com/pypa/pip/commit/d051a00fc57037104fca85ad8ebf2cdbd1e32d24#diff-058e40cb3a9ea705f655937e48f3a053f5dc7c500b7f1b2aae76e9bd673faf64

so a solution, maybe this patch, is needed to keep pipdeptree alive

[Midnighter]

As a note, I've moved the functionality that I care about to https://github.com/Midnighter/dependency-info. It's based entirely on importlib metadata. It only has a very minimal feature set, though.

[gaborbernat]

Closing as this seems to have stalled.