Doesn't work with extras_require and tests_require

[dvorapa]

I was a bit surprised, that pipdeptree can not handle either extra, or test dependencies. See for example: Expected:

$ python -mpipdeptree
pipdeptree==0.13.0
  - pip [required: >=6.0.0, installed: 18.0]
pywikibot==3.0.dev0
  - crontab==0.22.2
  - flickrapi==2.4.0
    - requests [required: >=2.2.1, installed: 2.19.1]
      - chardet [required: >=3.0.2, installed: 3.0.4]
      - idna [required: >=2.5, installed: 2.7]
      - urllib3 [required: >=1.21.1, installed: 1.23]
    - requests-oauthlib [required: >=0.4.0, installed: 1.0.0]
      - oauthlib [required: >=0.6.2, installed: 2.1.0]
      - requests [required: >=2.0.0, installed: 2.19.1]
        - chardet [required: >=3.0.2, installed: 3.0.4]
        - idna [required: >=2.5, installed: 2.7]
        - urllib3 [required: >=1.21.1, installed: 1.23]
    - requests-toolbelt [required: >=0.3.1, installed: 0.8.0]
      - requests [required: >=2.0.1,<3.0.0, installed: 2.19.1]
        - chardet [required: >=3.0.2, installed: 3.0.4]
        - idna [required: >=2.5, installed: 2.7]
        - urllib3 [required: >=1.21.1, installed: 1.23]
    - six [required: >=1.5.2, installed: 1.11.0]
  - google==2.0.1
    - beautifulsoup4 [required: Any, installed: 4.6.3]
  - irc==16.4
    - jaraco.collections [required: Any, installed: 1.5.3]
      - jaraco.classes [required: Any, installed: 1.5]
        - six [required: Any, installed: 1.11.0]
      - jaraco.text [required: Any, installed: 1.10.1]
        - jaraco.functools [required: Any, installed: 1.20]
          - more-itertools [required: Any, installed: 4.3.0]
            - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
      - six [required: >=1.7.0, installed: 1.11.0]
    - jaraco.functools [required: >=1.20, installed: 1.20]
      - more-itertools [required: Any, installed: 4.3.0]
        - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
    - jaraco.itertools [required: >=1.8, installed: 2.5.2]
      - inflect [required: Any, installed: 1.0.1]
      - more-itertools [required: >=4.0.0, installed: 4.3.0]
        - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
      - six [required: Any, installed: 1.11.0]
    - jaraco.logging [required: Any, installed: 1.5.2]
      - six [required: Any, installed: 1.11.0]
      - tempora [required: Any, installed: 1.13]
        - jaraco.functools [required: >=1.20, installed: 1.20]
          - more-itertools [required: Any, installed: 4.3.0]
            - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
        - pytz [required: Any, installed: 2018.5]
        - six [required: Any, installed: 1.11.0]
    - jaraco.stream [required: Any, installed: 1.2]
      - six [required: Any, installed: 1.11.0]
    - jaraco.text [required: Any, installed: 1.10.1]
      - jaraco.collections [required: Any, installed: 1.5.3]
        - jaraco.classes [required: Any, installed: 1.5]
          - six [required: Any, installed: 1.11.0]
        - six [required: >=1.7.0, installed: 1.11.0]
      - jaraco.functools [required: Any, installed: 1.20]
        - more-itertools [required: Any, installed: 4.3.0]
          - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
    - more-itertools [required: Any, installed: 4.3.0]
      - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
    - pytz [required: Any, installed: 2018.5]
    - six [required: Any, installed: 1.11.0]
    - tempora [required: >=1.6, installed: 1.13]
      - jaraco.functools [required: >=1.20, installed: 1.20]
        - more-itertools [required: Any, installed: 4.3.0]
          - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
      - pytz [required: Any, installed: 2018.5]
      - six [required: Any, installed: 1.11.0]
  - memento-client==0.6.1
    - requests [required: >=2.7.0, installed: 2.19.1]
      - chardet [required: >=3.0.2, installed: 3.0.4]
      - idna [required: >=2.5, installed: 2.7]
      - urllib3 [required: >=1.21.1, installed: 1.23]
  - mwoauth==0.3.2
    - oauthlib [required: Any, installed: 2.1.0]
    - PyJWT [required: >=1.0.1,<2.0.0, installed: 1.6.4]
    - requests [required: Any, installed: 2.19.1]
      - chardet [required: >=3.0.2, installed: 3.0.4]
      - idna [required: >=2.5, installed: 2.7]
      - urllib3 [required: >=1.21.1, installed: 1.23]
    - requests-oauthlib [required: Any, installed: 1.0.0]
      - oauthlib [required: >=0.6.2, installed: 2.1.0]
      - requests [required: >=2.0.0, installed: 2.19.1]
        - chardet [required: >=3.0.2, installed: 3.0.4]
        - idna [required: >=2.5, installed: 2.7]
        - urllib3 [required: >=1.21.1, installed: 1.23]
    - six [required: Any, installed: 1.11.0]
  - mwparserfromhell==0.5.1
  - Pillow==5.3.0
  - pycountry==18.5.26
  - pydot==1.2.4
    - pyparsing [required: >=2.1.4, installed: 2.2.2]
  - PyMySQL==0.9.2
    - cryptography [required: Any, installed: 2.3.1]
      - asn1crypto [required: >=0.21.0, installed: 0.24.0]
      - cffi [required: >=1.7,!=1.11.3, installed: 1.11.5]
        - pycparser [required: Any, installed: 2.19]
      - idna [required: >=2.1, installed: 2.7]
      - six [required: >=1.4.1, installed: 1.11.0]
  - python-stdnum==1.9
  - requests [required: >=2.9,!=2.18.2, installed: 2.19.1]
    - chardet [required: >=3.0.2, installed: 3.0.4]
    - idna [required: >=2.5, installed: 2.7]
    - urllib3 [required: >=1.21.1, installed: 1.23]
  - sseclient==0.0.19
    - requests [required: >=2.0.0, installed: 2.19.1]
      - chardet [required: >=3.0.2, installed: 3.0.4]
      - idna [required: >=2.5, installed: 2.7]
      - urllib3 [required: >=1.21.1, installed: 1.23]
    - six [required: Any, installed: 1.11.0]

Current:

$ python -mpipdeptree
crontab==0.22.2
flickrapi==2.4.0
  - requests [required: >=2.2.1, installed: 2.19.1]
    - chardet [required: >=3.0.2, installed: 3.0.4]
    - idna [required: >=2.5, installed: 2.7]
    - urllib3 [required: >=1.21.1, installed: 1.23]
  - requests-oauthlib [required: >=0.4.0, installed: 1.0.0]
    - oauthlib [required: >=0.6.2, installed: 2.1.0]
    - requests [required: >=2.0.0, installed: 2.19.1]
      - chardet [required: >=3.0.2, installed: 3.0.4]
      - idna [required: >=2.5, installed: 2.7]
      - urllib3 [required: >=1.21.1, installed: 1.23]
  - requests-toolbelt [required: >=0.3.1, installed: 0.8.0]
    - requests [required: >=2.0.1,<3.0.0, installed: 2.19.1]
      - chardet [required: >=3.0.2, installed: 3.0.4]
      - idna [required: >=2.5, installed: 2.7]
      - urllib3 [required: >=1.21.1, installed: 1.23]
  - six [required: >=1.5.2, installed: 1.11.0]
google==2.0.1
  - beautifulsoup4 [required: Any, installed: 4.6.3]
irc==16.4
  - jaraco.collections [required: Any, installed: 1.5.3]
    - jaraco.classes [required: Any, installed: 1.5]
      - six [required: Any, installed: 1.11.0]
    - jaraco.text [required: Any, installed: 1.10.1]
      - jaraco.functools [required: Any, installed: 1.20]
        - more-itertools [required: Any, installed: 4.3.0]
          - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
    - six [required: >=1.7.0, installed: 1.11.0]
  - jaraco.functools [required: >=1.20, installed: 1.20]
    - more-itertools [required: Any, installed: 4.3.0]
      - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
  - jaraco.itertools [required: >=1.8, installed: 2.5.2]
    - inflect [required: Any, installed: 1.0.1]
    - more-itertools [required: >=4.0.0, installed: 4.3.0]
      - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
    - six [required: Any, installed: 1.11.0]
  - jaraco.logging [required: Any, installed: 1.5.2]
    - six [required: Any, installed: 1.11.0]
    - tempora [required: Any, installed: 1.13]
      - jaraco.functools [required: >=1.20, installed: 1.20]
        - more-itertools [required: Any, installed: 4.3.0]
          - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
      - pytz [required: Any, installed: 2018.5]
      - six [required: Any, installed: 1.11.0]
  - jaraco.stream [required: Any, installed: 1.2]
    - six [required: Any, installed: 1.11.0]
  - jaraco.text [required: Any, installed: 1.10.1]
    - jaraco.collections [required: Any, installed: 1.5.3]
      - jaraco.classes [required: Any, installed: 1.5]
        - six [required: Any, installed: 1.11.0]
      - six [required: >=1.7.0, installed: 1.11.0]
    - jaraco.functools [required: Any, installed: 1.20]
      - more-itertools [required: Any, installed: 4.3.0]
        - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
  - more-itertools [required: Any, installed: 4.3.0]
    - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
  - pytz [required: Any, installed: 2018.5]
  - six [required: Any, installed: 1.11.0]
  - tempora [required: >=1.6, installed: 1.13]
    - jaraco.functools [required: >=1.20, installed: 1.20]
      - more-itertools [required: Any, installed: 4.3.0]
        - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
    - pytz [required: Any, installed: 2018.5]
    - six [required: Any, installed: 1.11.0]
memento-client==0.6.1
  - requests [required: >=2.7.0, installed: 2.19.1]
    - chardet [required: >=3.0.2, installed: 3.0.4]
    - idna [required: >=2.5, installed: 2.7]
    - urllib3 [required: >=1.21.1, installed: 1.23]
mwoauth==0.3.2
  - oauthlib [required: Any, installed: 2.1.0]
  - PyJWT [required: >=1.0.1,<2.0.0, installed: 1.6.4]
  - requests [required: Any, installed: 2.19.1]
    - chardet [required: >=3.0.2, installed: 3.0.4]
    - idna [required: >=2.5, installed: 2.7]
    - urllib3 [required: >=1.21.1, installed: 1.23]
  - requests-oauthlib [required: Any, installed: 1.0.0]
    - oauthlib [required: >=0.6.2, installed: 2.1.0]
    - requests [required: >=2.0.0, installed: 2.19.1]
      - chardet [required: >=3.0.2, installed: 3.0.4]
      - idna [required: >=2.5, installed: 2.7]
      - urllib3 [required: >=1.21.1, installed: 1.23]
  - six [required: Any, installed: 1.11.0]
mwparserfromhell==0.5.1
Pillow==5.3.0
pipdeptree==0.13.0
  - pip [required: >=6.0.0, installed: 18.0]
pycountry==18.5.26
pydot==1.2.4
  - pyparsing [required: >=2.1.4, installed: 2.2.2]
PyMySQL==0.9.2
  - cryptography [required: Any, installed: 2.3.1]
    - asn1crypto [required: >=0.21.0, installed: 0.24.0]
    - cffi [required: >=1.7,!=1.11.3, installed: 1.11.5]
      - pycparser [required: Any, installed: 2.19]
    - idna [required: >=2.1, installed: 2.7]
    - six [required: >=1.4.1, installed: 1.11.0]
python-stdnum==1.9
pywikibot==3.0.dev0
  - requests [required: >=2.9,!=2.18.2, installed: 2.19.1]
    - chardet [required: >=3.0.2, installed: 3.0.4]
    - idna [required: >=2.5, installed: 2.7]
    - urllib3 [required: >=1.21.1, installed: 1.23]
sseclient==0.0.19
  - requests [required: >=2.0.0, installed: 2.19.1]
    - chardet [required: >=3.0.2, installed: 3.0.4]
    - idna [required: >=2.5, installed: 2.7]
    - urllib3 [required: >=1.21.1, installed: 1.23]
  - six [required: Any, installed: 1.11.0]

[naiquevin]

Hi, I didn't get what you exactly mean by "handle" extra/test dependencies. It would be helpful if you could please add just the line diff between expected and actual output.

[dvorapa]

Hi, I didn't get what you exactly mean by "handle" extra/test dependencies. It would be helpful if you could please add just the line diff between expected and actual output.

The example shows 20 packages. But 18 of them were installed using pip install -e pywikibot[extras] (or pip install -r requirements.txt) (as extra dependencies of pywikibot) and therefore should be marked as dependencies of pywikibot. They are not. They are marked as explicitly installed packages, which they are not. extras_require and test_require are parts of pip specification, both containing dependencies similar to install_require (which contains the dependencies pipdeptree can work with).

This issue is a huge limitation as it does not mark dependencies as dependencies and therefore the tree is not correct.

[naiquevin]

Ok, got it now. Thanks for the clarification.

pipdeptree uses pip's internals to build the graph. Will have to check if pip exposes additional metadata for extras and test dependencies. Only in that case, it can be supported.

[AWhetter]

The DistInfoDistribution.requires() (the class that pip returns from get_installed_distributions()) takes an extras argument to specify which extras to query for requirements.

from pip._internal.utils.misc import get_installed_distributions
help(get_installed_distributions()[13].requires)

Each Requirement object returned from requires() has an extras that says which extras the requirement comes from.

get_installed_distributions()[13].requires()[2].extras

It should be possible to use these to store extras information in the tree somehow.

For a smaller test case, installing jira gives the following deptree:

cryptography==2.7
  - asn1crypto [required: >=0.21.0, installed: 0.24.0]
  - cffi [required: >=1.8,!=1.11.3, installed: 1.12.3]
    - pycparser [required: Any, installed: 2.19]
  - enum34 [required: Any, installed: 1.1.6]
  - ipaddress [required: Any, installed: 1.0.22]
  - six [required: >=1.4.1, installed: 1.12.0]
jira==2.0.0
  - defusedxml [required: Any, installed: 0.6.0]
  - oauthlib [required: >=1.0.0, installed: 3.0.2]
  - pbr [required: >=3.0.0, installed: 5.4.1]
  - requests [required: >=2.10.0, installed: 2.22.0]
    - certifi [required: >=2017.4.17, installed: 2019.6.16]
    - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
    - idna [required: >=2.5,<2.9, installed: 2.8]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.3]
  - requests-oauthlib [required: >=0.6.1, installed: 1.2.0]
    - oauthlib [required: >=3.0.0, installed: 3.0.2]
    - requests [required: >=2.0.0, installed: 2.22.0]
      - certifi [required: >=2017.4.17, installed: 2019.6.16]
      - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
      - idna [required: >=2.5,<2.9, installed: 2.8]
      - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.3]
  - requests-toolbelt [required: Any, installed: 0.9.1]
    - requests [required: >=2.0.1,<3.0.0, installed: 2.22.0]
      - certifi [required: >=2017.4.17, installed: 2019.6.16]
      - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
      - idna [required: >=2.5,<2.9, installed: 2.8]
      - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.3]
  - setuptools [required: >=20.10.1, installed: 41.0.1]
  - six [required: >=1.10.0, installed: 1.12.0]
pipdeptree==0.13.2
  - pip [required: >=6.0.0, installed: 19.2.1]
PyJWT==1.7.1
wheel==0.33.4

Jira actually depends on oauthlib[signedtoken] (ie oauthlib with the signedtoken extra), which depends on cryptography and PyJWT. Because pipdeptree does not detect either as a dependency of the oauthlib extra, they are incorrectly listed as top level packages.

[naiquevin]

@AWhetter: This is helpful. Thanks!

[wimglenn]

@naiquevin Note that since Python 3.8 you can use importlib.metadata to get this "extra" info easily:

>>> from importlib.metadata import requires
>>> requires("jira")
['defusedxml', 'oauthlib[signedtoken] (>=1.0.0)', 'pbr (>=3.0.0)', 'requests-oauthlib (>=0.6.1)', 'requests (>=2.10.0)', 'requests-toolbelt', 'setuptools (>=20.10.1)', 'six (>=1.10.0)', "argparse; (python_version<'2.7')", "requests-futures (>=0.9.7); extra == 'async'", 'ipython (<6.0.0,>=4.0.0); python_version < "3.3" and extra == \'cli\'', 'ipython (>=4.0.0); python_version >= "3.3" and extra == \'cli\'', "filemagic (>=1.6); extra == 'opt'", "PyJWT; extra == 'opt'", "requests-jwt; extra == 'opt'", "requests-kerberos; extra == 'opt'"]
>>> requires("oauthlib")
["cryptography ; extra == 'rsa'", "blinker ; extra == 'signals'", "cryptography ; extra == 'signedtoken'", "pyjwt (>=1.0.0) ; extra == 'signedtoken'"]

@AWhetter pip._internal should not be imported. It's private API.

[flying-sheep]

Is there an alternative that supports this?

[flying-sheep]

We can use graphviz’ shape=record for this. One real life example (simplified):

digraph structs {
    node [shape=record]
    edge [color="#00000077"]
    
    api    [label="{ <base> api    | { <test> test | <dev> dev | <testing> testing }}"]
    server [label="{ <base> server | { <test> test | <dev> dev | <testing> testing }}"]
    utils  [label="{ <base> utils  | { <test> test | <dev> dev }}"]
    pytest [label="<base> pytest"]

    api:base -> utils:base
    server:base -> utils:base

    api:testing -> server:testing
    utils:test -> pytest:base
    api:test -> pytest:base
    server:test -> pytest:base
    server:testing -> pytest:base
}

Here, each packages’ tests depend on pytest, whereas both api and server have a extra called testing (containing test utils that other packages can use).

pipdeptree just ignores that api has an optional dependency on server, leaving out the actually important information I would have needed to debug a nasty dependency issue. I would have needed an output like this:

$ pipdeptree -p api[testing] --graph-output=dot | xdot /dev/stdin

[naiquevin]

There's an open PR for supporting extra requirements - #138 , which basically builds on top of another PR #132. I haven't been able to spend much time on it after the initial work. IIRC, most changes are done but I don't want to haste the release of it as there's a chance of introducing regressions.

Let me rebase that branch with latest master over this weekend so that it's at least usable on experimental basis.

[dill0wn]

I just ran into this issue today. My use case is actually in a python script, so i'll take a look at importlib.metadata as suggested in the earlier comment https://github.com/naiquevin/pipdeptree/issues/107#issuecomment-564457047

EDIT: Nevermind, I'm still on 3.7 👎 so I can't.

[flying-sheep]

You can use importlib_metadata, the backport on PyPI

[ralphie0112358]

Do you plan to fix this issue?

[naiquevin]

I doubt if I can get to this anytime soon due to other commitments. The code is there on another branch but master branch has deviated now. And the changes are not trivial so it will take some time and effort to rebase.

On Tue, 1 Mar 2022 at 6:41 AM, ralphie0112358 @.***> wrote:

Do you plan to fix this issue?

— Reply to this email directly, view it on GitHub https://github.com/naiquevin/pipdeptree/issues/107#issuecomment-1054873060, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEOZHHCB5PSRDAHX5IBYXDU5QML3ANCNFSM4FZREJDA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

-- ~ Vineet

[gaborbernat]

A PR trying to address this would be welcome! tests_require is not supported but extra should be.

[Kagami]

There's similar issue with jupyterlab: one of its dependencies jupyter_events has jsonschema[format-nongpl] as dependency which depends on:

format-nongpl = [
  "fqdn",
  "idna",
  "isoduration",
  "jsonpointer>1.13",
  "rfc3339-validator",
  "rfc3986-validator>0.1.0",
  "uri_template",
  "webcolors>=1.11",
]

so you have a lot of extra top-level dependencies in the output:

$ pipdeptree -u -d 0
fqdn==1.5.1
isoduration==20.11.0
jsonpointer==2.4
jupyterlab==4.0.2
uri-template==1.3.0
webcolors==1.13

Note that only jupyterlab here is the only real top-level dependency.