Pip virtual environments unexpected system-level vulns

[Lewiscowles1986]

Bug description

Caveat: I May be mistaken, and this may be an intentional way the system operates. Operating under the principle of least surprise, I've classified as a bug. It may be a feature request.

I expect that using pip-audit -l inside a virtual environment, would only audit packages output by pip freeze. This is not the case. I Think it should be the case.

Reproduction steps

python -m pip install virtualenv
python -m virtualenv .venv
. .venv/Scripts/activate
pip install pip-audit
pip freeze > requirements.txt
pip-audit -r ./requirements.txt
pip-audit -l

Expected behavior

I Believe pip-audit -l should work similarly to pip freeze > /tmp/{noise}-requirements.txt && pip-audit -r /tmp/{noise}-requirements.txt

Screenshots and logs

Requirements.txt

pip-freeze==1.0.0

Output

pip-audit -l (abbreviated output)

Found 32 known vulnerabilities in 18 packages
Name         Version ID             Fix Versions
------------ ------- -------------- -------------------
babel        2.8.0   PYSEC-2021-421 2.9.1
cryptography 2.8     PYSEC-2021-62  3.2.1

pip audit -r ./requirements.txt

No known vulnerabilities found

Platform information

• OS name and version: Windows 10 64-bit Pro
• pip-audit version (pip-audit -V): 1.0.0
• Python version (python -V or python3 -V): 3.10.0
• pip version (pip -V or pip3 -V): 21.3.1

Additional context

Add any other context about the problem here.

[tetsuo-cpp]

Thanks for the report @Lewiscowles1986.

The -l flag is meant to be modelled after pip list -l which filters out system packages when being run within a virtual environment. I think the issue here is that our help text for the -l flag is vague and doesn't explicitly refer to a virtual environment.

It'd be better if I reworded it to match the pip list -h output like:

-l, --local                 If in a virtualenv that has global access, do not audit globally-installed packages

Does that seem reasonable to you?

[tetsuo-cpp]

My bad, I parsed this wrong. You're right, I wouldn't expect pip-audit to pick up globally installed packages with the -l flag.

Let me try to reproduce this myself.

[tetsuo-cpp]

@woodruffw I can't seem to reproduce this, could you give it a try? In order for my virtualenv to "inherit" the system packages, I need to create it with --system-site-packages.

At that point, the pip-audit -l seems to work as expected. The other weird thing to me is that if you do pip freeze > requirements.txt as described in the issue, you'd actually expect to have the system packages written to the requirements file since the -l flag isn't being used.

@Lewiscowles1986 We use pip-api to get dependencies from the environment, which uses pip list under the hood here. Does pip list and pip list -l give you the results that you'd expect (the system packages shouldn't appear when using the -l flag)?

[di]

I'm also having a hard time reproducing/understanding the original issue, but I think the key is that the equivalent of pip-audit -l would be pip freeze -l > requirements.txt && pip-audit -r requirements.txt -- note the -l flag to pip freeze. Otherwise, you'll get globally installed packages if the virtualenv has global access.

The closest I came to reproducing is below, but this is working as expected as far as I can tell:

$ docker run -it python:3.9 bash

root@da4021ed7bce:/# pip install cryptography==2.8
Collecting cryptography==2.8
...
Successfully installed cffi-1.15.0 cryptography-2.8 pycparser-2.21 six-1.16.0

root@da4021ed7bce:/# python -m venv env --system-site-packages

root@da4021ed7bce:/# source env/bin/activate

(env) root@da4021ed7bce:/# pip list
Package      Version
------------ -------
cffi         1.15.0
cryptography 2.8
pip          21.2.4
pycparser    2.21
setuptools   58.1.0
six          1.16.0
wheel        0.37.0

(env) root@da4021ed7bce:/# pip list -l
Package    Version
---------- -------
pip        21.2.4
setuptools 58.1.0

(env) root@da4021ed7bce:/# pip freeze
cffi==1.15.0
cryptography==2.8
pycparser==2.21
six==1.16.0

(env) root@da4021ed7bce:/# pip freeze -l

(env) root@da4021ed7bce:/# pip install pip-audit
Collecting pip-audit
...
Successfully installed CacheControl-0.12.10 certifi-2021.10.8 charset-normalizer-2.0.8 cyclonedx-python-lib-0.11.1 html5lib-1.1 idna-3.3 lockfile-0.12.2 msgpack-1.0.3 packageurl-python-0.9.6 packaging-21.3 pip-api-0.0.23 pip-audit-1.0.0 progress-1.6 pyparsing-3.0.6 requests-2.26.0 requirements-parser-0.2.0 resolvelib-0.8.1 setuptools-50.3.2 toml-0.10.2 types-setuptools-57.4.4 types-toml-0.10.1 urllib3-1.26.7 webencodings-0.5.1

(env) root@da4021ed7bce:/# pip-audit
\ Auditing wheel (0.37.0)
Found 1 known vulnerabilities in 1 packages
Name         Version ID            Fix Versions
------------ ------- ------------- ------------
cryptography 2.8     PYSEC-2021-62 3.2.1

(env) root@da4021ed7bce:/# pip-audit -l

- Auditing webencodings (0.5.1)
No known vulnerabilities found

[Lewiscowles1986]

Hmm

lewis@LAPTOP-DCH0M5G9 MINGW64 ~/projects/labs/python/py-audit
$ pip list
Package              Version
-------------------- ---------
CacheControl         0.12.10
certifi              2021.10.8
charset-normalizer   2.0.8
cyclonedx-python-lib 0.11.1
html5lib             1.1
idna                 3.3
lockfile             0.12.2
msgpack              1.0.3
packageurl-python    0.9.6
packaging            21.3
pip                  21.3.1
pip-api              0.0.23
pip-audit            1.0.0
progress             1.6
pyparsing            3.0.6
requests             2.26.0
requirements-parser  0.2.0
resolvelib           0.8.1
setuptools           50.3.2
six                  1.16.0
toml                 0.10.2
types-setuptools     57.4.4
types-toml           0.10.1
urllib3              1.26.7
webencodings         0.5.1
wheel                0.37.0

(.venv)
lewis@LAPTOP-DCH0M5G9 MINGW64 ~/projects/labs/python/py-audit
$ pip list -l
Package              Version
-------------------- ---------
CacheControl         0.12.10
certifi              2021.10.8
charset-normalizer   2.0.8
cyclonedx-python-lib 0.11.1
html5lib             1.1
idna                 3.3
lockfile             0.12.2
msgpack              1.0.3
packageurl-python    0.9.6
packaging            21.3
pip                  21.3.1
pip-api              0.0.23
pip-audit            1.0.0
progress             1.6
pyparsing            3.0.6
requests             2.26.0
requirements-parser  0.2.0
resolvelib           0.8.1
setuptools           50.3.2
six                  1.16.0
toml                 0.10.2
types-setuptools     57.4.4
types-toml           0.10.1
urllib3              1.26.7
webencodings         0.5.1
wheel                0.37.0

[Lewiscowles1986]

Looking at the requirements.txt it was definitely wrong what I shared yesterday. pip-freeze==1.0.0 as the only requirement...

[richardash1981]

I have had pretty much the same experience trying to get pip-audit to function on Ubuntu Linux, with system packages (and the system versions of packages which are updated in my virtual environment) leaking into the results for virtual environments. I'm not sure why this happens, but I have suspicions about the shebang (first line) of the pip-audit script itself.

Anyway, what works in Ubuntu 18.04 and 20.04 is as follows:

• Don't try to install pip-audit in the Ubuntu environment, even using pip
• Create an empty virtual environment
• Install just pip-audit in the environment
• Audit as required from within the (nearly empty) virtual environment.

python3 -m venv .venv
source .venv/bin/activate
pip install pip-audit
pip-audit --strict -r requirements.txt

Inserting the extra layer between the OS and pip-audit seems to be necessary to get rid of this interference from the Ubuntu packaging. This is something to do with how Python is installed, because installing pip-audit outside of a venv (using pip --user) and then executing it with the requirements file or within a virtual environment works fine on Gentoo Linux (where a lot of effort is put in to good Python support by the distro team - presumably this is it showing).

Regarding the --system-site-packages option, I have confirmed that this option is off by default (saved as such in //pyvenv.cfg// within the venv).

$ python3 -m venv .venv
$ . ./.venv/bin/activate
$ pip list
pip (9.0.1)
pkg-resources (0.0.0)
setuptools (39.0.1)
$ pip install pip-audit
[...]
$ pip list
CacheControl (0.12.11)
certifi (2022.5.18.1)
charset-normalizer (2.0.12)
cyclonedx-python-lib (0.12.3)
dataclasses (0.8)
html5lib (1.1)
idna (3.3)
importlib-metadata (4.8.3)
lockfile (0.12.2)
msgpack (1.0.3)
packageurl-python (0.9.9)
packaging (21.3)
pip (9.0.1)
pip-api (0.0.26)
pip-audit (1.1.2)
pkg-resources (0.0.0)
progress (1.6)
pyparsing (3.0.9)
requests (2.27.1)
resolvelib (0.8.1)
setuptools (59.6.0)
six (1.16.0)
toml (0.10.2)
types-setuptools (57.4.17)
types-toml (0.10.7)
typing-extensions (3.10.0.2)
urllib3 (1.26.9)
webencodings (0.5.1)
zipp (3.6.0)

This all looks OK (but note pkg-resources (0.0.0) is present). All the other packages were installed by pip into the virtual environment so are correct.

[woodruffw]

I'm not sure why this happens, but I have suspicions about the shebang (first line) of the pip-audit script itself.

pip-audit's shebang is generated during pip install, using the standard "entrypoints" technique. I don't think that's (directly) the source of the problem here, except for in the sense that it references your system's default Python and user package environment (which are broken because of how Ubuntu/Debian package Python; see below).

This is something to do with how Python is installed, because installing pip-audit outside of a venv (using pip --user) and then executing it with the requirements file or within a virtual environment works fine on Gentoo Linux (where a lot of effort is put in to good Python support by the distro team - presumably this is it showing).

Yep. It's because Ubuntu debundles core parts of Python. Examples include python3-pip and python3-distutils, which are normally (and correctly) part of the Python distribution itself. This has lots of other negative consequences, like the extremely old pip version shown in your pip list.

It might not help in your use case, but in general Python developers/deployers are encouraged to use virtual environments everywhere and to never rely on the system/user package environment. If you can do that with your project/deployment, you'll probably run into a lot less problems (not just with pip-audit, but also with pip and most other packaging tools).

[woodruffw]

Tagging as upstream to emphasize that this is not a bug in pip-audit per se, but a consequence of Ubuntu/Debian's packaging decisions.

[Lewiscowles1986]

I Stopped using this just as soon as I'd investigated it. It's totally fine, but I'm unsubscribing now. Thanks.

[woodruffw]

Hmm, I was able to reproduce a version of this on macOS.

I have a virtual environment with some packages installed:

Package           Version
----------------- -------
black             22.3.0
blight            0.0.47
click             8.1.3
flake8            4.0.1
isort             5.10.1
mccabe            0.6.1
mypy-extensions   0.4.3
pathspec          0.9.0
pip               22.1.2
platformdirs      2.5.2
pycodestyle       2.8.0
pydantic          1.9.1
pyflakes          2.4.0
setuptools        57.4.0
tomli             2.0.1
typing_extensions 4.2.0

and I have pip-audit installed at the user level via a pyenv managed Python:

$ pip-audit --version
pip-audit 2.3.3

$ which pip-audit
/Users/william/.pyenv/shims/pip-audit

But when I run pip-audit --local, I get reports for things in my user environment, not the currently active virtual environment:

$ pip-audit --local
Found 1 known vulnerability in 1 package
Name  Version ID             Fix Versions
----- ------- -------------- ------------
pyjwt 2.3.0   PYSEC-2022-202 2.4.0
Name                 Skip Reason
-------------------- -----------------------------------------------------------------------------------------
cryptography         Dependency not found on PyPI and could not be audited: cryptography (38.0.0.dev1)
cryptography-vectors Dependency not found on PyPI and could not be audited: cryptography-vectors (38.0.0.dev1)

Note that none of the things in the pip-audit output are installed in the currently active virtual environment.

And of course, since it's a virtual environment, the python -m entrypoint doesn't work:

$ python -m pip_audit
/Users/william/devel/.../env/bin/python: No module named pip_audit

So this looks like a bug on two fronts to me: we're not correctly collecting the virtual environment's packages (why?) and --local doesn't seem to have an effect (it maps directly to pip list --local, which is presumably a no-op if it's escaping the virtual environment.)

[woodruffw]

Okay, this is probably it, in pip-api:

def call(*args, cwd=None):
    python_location = os.environ.get("PIPAPI_PYTHON_LOCATION", sys.executable)
    env = {**os.environ, **{"PIP_YES": "true", "PIP_DISABLE_PIP_VERSION_CHECK": "true"}}
    result = subprocess.check_output(
        [python_location, "-m", "pip"] + list(args), cwd=cwd, env=env
    )
    return result.decode()

We use sys.executable as the fallback for location an appropriate python for python -m pip, which punctures the virtual environment because pip-audit itself is not installed in the venv.

So, I think we have two options:

1. Create an error case here. In particular, if we detect that we're running under a virtual environment and that our sys.executable doesn't match the venv's we could fail and ask the user to install pip-audit into the virtual environment instead of globally.
   • Pros: More correct (maybe?)
   • Cons: Requires users to think about where pip-audit is installed, which exposes internal complexity and isn't consistent with other QA tools like black (which work just fine regardless if they're installed outside of the current venv)
2. Detect that we're in a virtual environment and set PIPAPI_PYTHON_LOCATION to compensate, causing pip-api to execute the correct python -m pip ....
   • Pros: Less user complexity, works the way users expect.
   • Cons: Is this sound? I have no idea what the implications are for installing a program outside of a venv, running it inside of one, but pretending to some parts of it that it's actually installed within the venv.

cc @di for thoughts.

Finally, I wanted to offer an apology to @Lewiscowles1986: I dismissed your theory about the shebang, and I was wrong (it's not the shebang itself that's causing the problem, but it indicates the environment mismatch that eventually confuses pip-api). This is 100% a bug in pip-audit.

[woodruffw]

One minor complexity: import pip_api has the side effect of running pip --version, which means that we need to ensure that we always set the PIPAPI_PYTHON_VERSION environment variable before pip_api is imported.

This shouldn't be a problem for the pip-audit CLI though, since we don't import pip_api anywhere in it (it only happens later, as an implementation detail of the audit API.)