pipdeptree compatibility issue with pip=21.3

[jpvandervelden]

pipdeptree used to run fine with the latest version of pip. Since Oct 11th pip version 21.3 is out and it broke the pipdeptree.

#14 [11/18] RUN pipdeptree #14 sha256:7c8f692e9903fb11ca8bb2f9cc2051d6f3fa68285ed3b7caaa0b3a349fa37796 #14 0.638 Traceback (most recent call last): #14 0.638 File "/usr/local/lib/python3.8/dist-packages/pipdeptree.py", line 25, in <module> #14 0.638 from pip._internal.utils.misc import get_installed_distributions #14 0.638 ImportError: cannot import name 'get_installed_distributions' from 'pip._internal.utils.misc' (/usr/local/lib/python3.8/dist-packages/pip/_internal/utils/misc.py) #14 0.638 #14 0.638 During handling of the above exception, another exception occurred: #14 0.638 #14 0.638 Traceback (most recent call last): #14 0.638 File "/usr/local/bin/pipdeptree", line 5, in <module> #14 0.638 from pipdeptree import main #14 0.638 File "/usr/local/lib/python3.8/dist-packages/pipdeptree.py", line 28, in <module> #14 0.638 from pip import get_installed_distributions, FrozenRequirement #14 0.638 ImportError: cannot import name 'get_installed_distributions' from 'pip' (/usr/local/lib/python3.8/dist-packages/pip/__init__.py) #14 ERROR: executor failed running [/bin/sh -c pipdeptree]: exit code: 1 ------

[naiquevin]

There is an open PR that fixes it - https://github.com/naiquevin/pipdeptree/pull/154.

I thought there was 1 more week before 21.3 is released. Let me see if a new version can be released for this.

[naiquevin]

Released pipdeptree version 2.2.0 with the fix.

[rickhg12hs]

pipdeptree doesn't seem to honor --user-only now.

$ pipdeptree --version
2.2.0
$ python -m pip --version
pip 21.3 from /home/rick/.local/lib/python3.8/site-packages/pip (python 3.8)
$ python -m pip show ansible
Name: ansible
Version: 2.9.21
Summary: Radically simple IT automation
Home-page: https://ansible.com/
Author: Ansible, Inc.
Author-email: [email protected]
License: GPLv3+
Location: /usr/lib/python3.8/site-packages
Requires: cryptography, jinja2, PyYAML
Required-by: 
$ pipdeptree --user-only -p ansible
ansible==2.9.21
  - cryptography [required: Any, installed: 3.3]
    - cffi [required: >=1.12, installed: 1.14.6]
      - pycparser [required: Any, installed: 2.20]
    - six [required: >=1.4.1, installed: 1.16.0]
  - jinja2 [required: Any, installed: 3.0.2]
    - MarkupSafe [required: >=2.0, installed: 2.0.1]
  - PyYAML [required: Any, installed: 5.4.1]

... and peeking at the changed code, I'm wondering if the --local-only option works too.

[jpvandervelden]

Works again! Thanks for the quick response!!

[fcastilloec]

@rickhg12hs I'm experiencing the same issue. I tried downgrading pipdeptree to 2.1.0 and use pip==21.3 and I got an error. The only working combination is pipdeptree==2.1.0 and pip==21.2.4

[naiquevin]

I haven't had the time to look at the changes in pip. It seems like --user-only and --local-only options can be supported (because pip still supports them). I am bit busy with other stuff at the moment, but I'll look at it soon.

[Steve-Z]

@rickhg12hs @fcastilloec I am also seeing failure of pipdeptree --user-only. Currently on pipdeptree==2.2.0 and pip==21.3.1. @naiquevin I appreciate all you do here. Please get to this when you can.

[naiquevin]

I've fixed the behaviour of --user-only and --freeze flags in latest commits on master. If any of you are using the --user-only flag, can you please test it against your env?

Will publish a new release sometime this week.

[rickhg12hs]

I installed via:

$ python -m pip install --upgrade --user --upgrade-strategy eager git+https://github.com/naiquevin/pipdeptree.git

... and it seems to mostly work, but I'm not sure it has the same behavior as before.

$ python -m pip --version
pip 21.3.1 from /home/rick/.local/lib/python3.8/site-packages/pip (python 3.8)
$ python -m pip show ansible
Name: ansible
Version: 2.9.21
Summary: Radically simple IT automation
Home-page: https://ansible.com/
Author: Ansible, Inc.
Author-email: [email protected]
License: GPLv3+
Location: /usr/lib/python3.8/site-packages
Requires: cryptography, jinja2, PyYAML
Required-by: 
$ pipdeptree --user-only -p ansible
Warning!!! Possibly conflicting dependencies found:
* bleach==4.1.0
 - webencodings [required: Any, installed: ?]
* Sphinx==4.3.2
 - alabaster [required: >=0.7,<0.8, installed: ?]
 - sphinxcontrib-jsmath [required: Any, installed: ?]
------------------------------------------------------------------------

$ pipdeptree -p ansible
ansible==2.9.21
  - cryptography [required: Any, installed: 3.3]
    - cffi [required: >=1.12, installed: 1.15.0]
      - pycparser [required: Any, installed: 2.21]
    - six [required: >=1.4.1, installed: 1.16.0]
  - jinja2 [required: Any, installed: 3.0.3]
    - MarkupSafe [required: >=2.0, installed: 2.0.1]
  - PyYAML [required: Any, installed: 5.4.1]
$ python -m pip check
No broken requirements found.

Did pipdeptree formerly warn about dependencies met through --user install?

This also doesn't seem quite right.

$ python -m pip show webencodings
Name: webencodings
Version: 0.5.1
Summary: Character encoding aliases for legacy web content
Home-page: https://github.com/SimonSapin/python-webencodings
Author: Simon Sapin
Author-email: [email protected]
License: BSD
Location: /usr/lib/python3.8/site-packages
Requires: 
Required-by: bleach, html5lib

$ pipdeptree -u -r -p webencodings
Warning!!! Possibly conflicting dependencies found:
* bleach==4.1.0
 - webencodings [required: Any, installed: ?]
* Sphinx==4.3.2
 - alabaster [required: >=0.7,<0.8, installed: ?]
 - sphinxcontrib-jsmath [required: Any, installed: ?]
------------------------------------------------------------------------
webencodings==?
  - bleach==4.1.0 [requires: webencodings]
    - nbconvert==6.3.0 [requires: bleach]
      - jupyter==1.0.0 [requires: nbconvert]
      - jupyter-tabnine==1.2.3 [requires: nbconvert]
      - notebook==6.4.6 [requires: nbconvert]
        - jupyter==1.0.0 [requires: notebook]
        - jupyter-tabnine==1.2.3 [requires: notebook>=4.2]
        - widgetsnbextension==3.5.2 [requires: notebook>=4.4.1]
          - ipywidgets==7.6.5 [requires: widgetsnbextension~=3.5.0]
            - jupyter==1.0.0 [requires: ipywidgets]
$ pipdeptree -r -p webencodings
webencodings==0.5.1
  - bleach==4.1.0 [requires: webencodings]
    - nbconvert==6.3.0 [requires: bleach]
      - jupyter==1.0.0 [requires: nbconvert]
      - jupyter-tabnine==1.2.3 [requires: nbconvert]
      - notebook==6.4.6 [requires: nbconvert]
        - jupyter==1.0.0 [requires: notebook]
        - jupyter-tabnine==1.2.3 [requires: notebook>=4.2]
        - widgetsnbextension==3.5.2 [requires: notebook>=4.4.1]
          - ipywidgets==7.6.5 [requires: widgetsnbextension~=3.5.0]
            - jupyter==1.0.0 [requires: ipywidgets]
  - html5lib==1.0.1 [requires: webencodings]
    - textile==4.0.1 [requires: html5lib>=1.0.1]

[naiquevin]

@rickhg12hs Thanks for testing it.

Did pipdeptree formerly warn about dependencies met through --user install?

To be honest, I am not sure. I will try running an older version of pipdeptree (with an old version of pip).

Would it be possible to upload the requirements.txt file of your env and a list of user-only packages?

[rickhg12hs]

@naiquevin My python reqs

Please let me know if you need more/different info.

[naiquevin]

@rickhg12hs Thanks

[naiquevin]

@rickhg12hs I could reproduce this issue by installing ansible with --user flag

vagrant@tmpvm:~/pipdeptree$ python3 pipdeptree.py -u -p ansible
defaultdict(<class 'list'>, {<DistPackage("ansible")>: [<ReqPackage("pyyaml")>]})
Warning!!! Possibly conflicting dependencies found:
* ansible==2.9.21
 - PyYAML [required: Any, installed: ?]
------------------------------------------------------------------------
ansible==2.9.21
  - cryptography [required: Any, installed: 2.1.4]
  - jinja2 [required: Any, installed: 2.10]
  - PyYAML [required: Any, installed: ?]
vagrant@tmpvm:~/pipdeptree$ python3 pipdeptree.py -u -p ansible,pyyaml
defaultdict(<class 'list'>, {<DistPackage("ansible")>: [<ReqPackage("pyyaml")>]})
Warning!!! Possibly conflicting dependencies found:
* ansible==2.9.21
 - PyYAML [required: Any, installed: ?]
------------------------------------------------------------------------
ansible==2.9.21
  - cryptography [required: Any, installed: 2.1.4]
  - jinja2 [required: Any, installed: 2.10]
  - PyYAML [required: Any, installed: ?]
vagrant@tmpvm:~/pipdeptree$ python3 pipdeptree.py -p ansible,pyyaml
defaultdict(<class 'list'>, {})
ansible==2.9.21
  - cryptography [required: Any, installed: 2.1.4]
  - jinja2 [required: Any, installed: 2.10]
  - PyYAML [required: Any, installed: 3.12]

I made a change to print the conflicts found before rendering the pipdeptree output.

Looks like with the -u flag the underlying pip function itself filters the packages ie. the filtering happens before checking for conflicts - and in this case, it's not able to find the version of PyYAML.

But with the -p flag, all packages are loaded into the tree data structure on which conflicts are checked. Then the filtering is done. In this case, the version of PyYAML is found.

To investigate further, I thought I'd try the following:

1. Uninstall ansible
2. Uninstall PyYAML
3. Reinstall ansible and PyYAML both with --user flag

But it doesn't allow me to uninstall PyYAML

Found existing installation: PyYAML 3.12
ERROR: Cannot uninstall 'PyYAML'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

I am guessing that PyYAML being a distutils project is what's causing it's version to be not found.

I think this is a package specific thing. I am proceeding with the release because the previous one is "more broken" 😅

[naiquevin]

Published version 2.2.1 to pypi

[rickhg12hs]

Yeah, I'm not really sure what the desired behavior should be. Should a tree be displayed if the root (or some leaves) don't exist (or represent) whats below it? ... or perhaps with added note that root or leaf is satisfied by some other means?

Agreed, release is better. 👍🏻

[Steve-Z]

@naiquevin Thank you! 2.2.1 fixed this.