pipenv icon indicating copy to clipboard operation
pipenv copied to clipboard
verified publisher icon pypa

Pipenv `graph` does not show dependencies installed to system when using `include-system-site-packages`

Open Askaholic opened this issue 2 years ago • 9 comments

I'm not 100% sure if this is intended behavior or considered a bug, so apologies if I made a mistake. I found some past issues with pipenv graph not including all dependencies in the output, however, none that were related to the interaction of the include-system-site-packages setting of the virtual environment.

Issue description

When creating virtual environments with the --site-packages option enabled, any dependencies that are present in the pipfile, but don't get installed into the virtual environment are omitted from the output of pipenv graph. However, they will still be included in the output of pipenv requirements.

Expected result

I would expect the output of pipenv graph to include all dependencies listed in the pipfile, similar to pipenv requirements.

Actual result

# pipenv --verbose graph
Running command: $ /root/.local/share/virtualenvs/root-BuDEOXnJ/bin/python /usr/local/lib/python3.8/site-packages/pipenv/vendor/pipdeptree -l
Command output: pip==22.3.1
setuptools==65.6.3
wheel==0.38.4

Steps to replicate

Pipfile:

[[source]]    
url = "https://pypi.org/simple"    
verify_ssl = true    
name = "pypi"    
    
[packages]    
idna = "==3.4"    
    
[dev-packages]    
    
[requires]    
python_version = "3.8"
  1. Copy the Pipfile to your working directory
  2. Install pipenv with sudo pip install pipenv==2023.2.4
  3. Install idna to system packages with sudo pip install idna==3.4
  4. Do pipenv install with pipenv install --site-packages
  5. Run pipenv graph

You will observe that no output is printed.

If you instead run pipenv requirements you will see:

-i https://pypi.org/simple
idna==3.4

I have also provided a Dockerfile to make it as easy as possible to reproduce the issue.

  1. Copy the Pipfile and the following Dockerfile into a directory:

Dockerfile:

FROM python:3.8

RUN pip install idna==3.4
RUN pip install pipenv==2023.2.4

COPY Pipfile /root/Pipfile
WORKDIR /root

RUN pipenv install --site-packages
  1. Build the docker image with docker build -t pipenv-test .
  2. Get a shell into the image with docker run --rm -it pipenv-test /bin/bash
  3. Run pipenv graph

You will observe that no output is printed.

If you instead run pipenv requirements you will see:

-i https://pypi.org/simple
idna==3.4
$ pipenv --support

Pipenv version: '2023.2.4'

Pipenv location: '/usr/local/lib/python3.8/site-packages/pipenv'

Python location: '/usr/local/bin/python'

OS Name: 'posix'

User pip version: '22.3.1'

user Python installations found:

  • 3.9.2: /usr/bin/python3.9
  • 3.9.2: /usr/bin/python3
  • 3.8.16: /usr/local/bin/python
  • 3.8.16: /usr/local/bin/python3.8
  • 3.8.16: /usr/local/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.8.16',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '5.14.0-1056-oem',
 'platform_system': 'Linux',
 'platform_version': '#63-Ubuntu SMP Fri Dec 16 14:32:59 UTC 2022',
 'python_full_version': '3.8.16',
 'python_version': '3.8',
 'sys_platform': 'linux'}

System environment variables:

  • HOSTNAME
  • PYTHON_VERSION
  • PWD
  • PYTHON_SETUPTOOLS_VERSION
  • HOME
  • LANG
  • GPG_KEY
  • TERM
  • SHLVL
  • PYTHON_PIP_VERSION
  • PYTHON_GET_PIP_SHA256
  • PYTHON_GET_PIP_URL
  • PATH
  • _
  • PIP_DISABLE_PIP_VERSION_CHECK
  • PIP_PYTHON_PATH
  • PYTHONDONTWRITEBYTECODE
  • PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

  • PATH: /usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
  • LANG: C.UTF-8
  • PWD: /root

Contents of Pipfile ('/root/Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
idna = "==3.4"

[dev-packages]

[requires]
python_version = "3.8"

Contents of Pipfile.lock ('/root/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "f37b9d7a3844a56b83b3df2a924934bf52170a3dd92c761782740b3f52516c72"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.8"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "idna": {
            "hashes": [
                "sha256:814f528e8dead7d329833b91c5faa87d60bf71824cd12a7530b5526063d02cb4",
                "sha256:90b77e79eaa3eba6de819a0c442c0b4ceefc341a7a2ab77d7562bf49f425c5c2"
            ],
            "index": "pypi",
            "version": "==3.4"
        }
    },
    "develop": {}
}

Askaholic avatar Feb 06 '23 18:02 Askaholic

@Askaholic I was not able to reproduce this issue, I used python 3.11 is the only difference in my test, and the latest pipenv.

matteius@matteius-VirtualBox:~/pipenv-triage$ mkdir issue-5597
matteius@matteius-VirtualBox:~/pipenv-triage$ cd issue-5597/
matteius@matteius-VirtualBox:~/pipenv-triage/issue-5597$ vim Pipfile
matteius@matteius-VirtualBox:~/pipenv-triage/issue-5597$ pipenv lock
Creating a virtualenv for this project...
Pipfile: /home/matteius/pipenv-triage/issue-5597/Pipfile
Using /home/matteius/.pyenv/versions/3.11.2/bin/python3 (3.11.2) to create virtualenv...
⠙ Creating virtual environment...created virtual environment CPython3.11.2.final.0-64 in 1078ms
  creator CPython3Posix(dest=/home/matteius/.virtualenvs/issue-5597-i00VrxRT, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/matteius/.local/share/virtualenv)
    added seed packages: pip==23.0, setuptools==67.1.0, wheel==0.38.4
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

✔ Successfully created virtual environment!
Virtualenv location: /home/matteius/.virtualenvs/issue-5597-i00VrxRT
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success!
Locking [dev-packages] dependencies...
Updated Pipfile.lock (4af47c6effaacb3d92fc9bb52a9daa525122b7bed0188c7d6bd39cac7a65db71)!
matteius@matteius-VirtualBox:~/pipenv-triage/issue-5597$ pipenv install --site-packages
Installing dependencies from Pipfile.lock (65db71)...
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
matteius@matteius-VirtualBox:~/pipenv-triage/issue-5597$ pipenv requirements
-i https://pypi.org/simple
idna==3.4
matteius@matteius-VirtualBox:~/pipenv-triage/issue-5597$ pipenv graph
idna==3.4

matteius avatar Feb 18 '23 22:02 matteius

@matteius I think you need to put the --site-packages flag on the first command that you run to create the virtual environment, in your case that would be the pipenv lock command. I’m not sure that pipenv will change the setting once your environment has been created.

Askaholic avatar Feb 18 '23 22:02 Askaholic

I also don’t see you installing the dependencies to the system there. The important thing is that when using site packages, the dependency won’t actually get installed into the virtual environment because it is already present in the system packages.

Askaholic avatar Feb 18 '23 22:02 Askaholic

@Askaholic Maybe, but I just retried and got similar results:

matteius@matteius-VirtualBox:~/pipenv-triage/issue-5597$ pipenv --rm
Removing virtualenv (/home/matteius/.virtualenvs/issue-5597-i00VrxRT)...
matteius@matteius-VirtualBox:~/pipenv-triage/issue-5597$ rm Pipfile*
matteius@matteius-VirtualBox:~/pipenv-triage/issue-5597$ vim Pipfile
matteius@matteius-VirtualBox:~/pipenv-triage/issue-5597$ pipenv --site-packages install
Creating a virtualenv for this project...
Pipfile: /home/matteius/pipenv-triage/issue-5597/Pipfile
Using /home/matteius/.pyenv/versions/3.8.12/bin/python3.8 (3.8.12) to create virtualenv...
Making site-packages available...
⠹ Creating virtual environment...created virtual environment CPython3.8.12.final.0-64 in 455ms
  creator CPython3Posix(dest=/home/matteius/.virtualenvs/issue-5597-i00VrxRT, clear=False, no_vcs_ignore=False, global=True)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/matteius/.local/share/virtualenv)
    added seed packages: pip==22.3, setuptools==65.5.1, wheel==0.38.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

✔ Successfully created virtual environment!
Virtualenv location: /home/matteius/.virtualenvs/issue-5597-i00VrxRT
Pipfile.lock not found, creating...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success!
Locking [dev-packages] dependencies...
Updated Pipfile.lock (f37b9d7a3844a56b83b3df2a924934bf52170a3dd92c761782740b3f52516c72)!
Installing dependencies from Pipfile.lock (516c72)...
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
matteius@matteius-VirtualBox:~/pipenv-triage/issue-5597$ pipenv graph
idna==3.4

matteius avatar Feb 18 '23 22:02 matteius

Did you run sudo pip install idna==3.4 before doing the pip install? I think doing a user install might have the same effect too.

Askaholic avatar Feb 18 '23 22:02 Askaholic

Sorry I thought that making Dockerfile would make things easier to reproduce but I guess not. My apologies.

Askaholic avatar Feb 18 '23 23:02 Askaholic

I can try it with the Dockerfile when I get more time -- on average it adds more time to reproducing an issue than it helps me with, but in this case it might be helpful.

matteius avatar Feb 18 '23 23:02 matteius

I updated the steps to reproduce to have just a list of commands to run. They are the same commands from the Dockerfile but written as an ordered list. Note that the order of steps 3 and 4 is very important. If you swap them, then you won't see the same result.

Askaholic avatar Feb 22 '23 15:02 Askaholic

We recently upgraded to the latest pipdeptree and it has the same behavior -- I asked GPT about this:

You're correct, pipdeptree only lists the packages and their dependencies installed in your virtual environment, not the ones from the system-wide site-packages. When you created your virtual environment with the --system-site-packages option, the system-wide packages become available to the virtual environment, but they are not listed as installed within the virtual environment itself.

However, these system packages are technically dependencies of your project if it needs them to run. You can find out which ones your project uses by running your project and seeing which imports it uses. Alternatively, you could use a tool such as pylint or pyflakes to statically analyze your code and find out which imports it uses.

If you want to include these packages as part of your virtual environment, you need to install them explicitly in the environment. Remember that this might defeat the purpose of the isolation that virtual environments provide. It can lead to version conflicts and other problems if different projects require different versions of the same package.

So I think if you want them to be displayed in graph, you would have to report this upstream to pipdeptree

matteius avatar Jun 11 '23 20:06 matteius