CalledProcessError when specifiying a package in requirements

[ccsalway]

Dont understand whats happening here.. I can run pip-audit against an installed pacakge, but naming the same package in the requriements.txt file renders and error...

% python3 -m venv --clear .venv && . .venv/bin/activate
% python -m pip install -Uq pip wheel setuptools pip-audit
% pip freeze
CacheControl==0.12.11
certifi==2022.6.15
charset-normalizer==2.1.0
commonmark==0.9.1
cyclonedx-python-lib==2.7.0
html5lib==1.1
idna==3.3
lockfile==0.12.2
msgpack==1.0.4
packageurl-python==0.10.0
packaging==21.3
pip-api==0.0.30
pip-requirements-parser==31.2.0
pip_audit==2.4.3
Pygments==2.12.0
pyparsing==3.0.9
requests==2.28.1
resolvelib==0.8.1
rich==12.5.1
six==1.16.0
sortedcontainers==2.4.0
toml==0.10.2
typing_extensions==4.3.0
urllib3==1.26.11
webencodings==0.5.1

% python -m pip_audit        
No known vulnerabilities found

% pip install -Uq flask==0.5
% python -m pip_audit   
Found 2 known vulnerabilities in 1 package
Name  Version ID             Fix Versions
----- ------- -------------- ------------
flask 0.5     PYSEC-2019-179 1.0
flask 0.5     PYSEC-2018-66  0.12.3

% echo "flask==0.5" > requirements.txt
% python -m pip_audit -r requirements.txt 
WARNING:cachecontrol.controller:Cache entry deserialization failed, entry ignored
Traceback (most recent call last):
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/private/var/folders/7b/w1fc1k292kgdj7g01ry8vf_40000gn/T/tmp.TBMsMgV1/.venv/lib/python3.8/site-packages/pip_audit/__main__.py", line 8, in <module>
    audit()
  File "/private/var/folders/7b/w1fc1k292kgdj7g01ry8vf_40000gn/T/tmp.TBMsMgV1/.venv/lib/python3.8/site-packages/pip_audit/_cli.py", line 428, in audit
    for (spec, vulns) in auditor.audit(source):
  File "/private/var/folders/7b/w1fc1k292kgdj7g01ry8vf_40000gn/T/tmp.TBMsMgV1/.venv/lib/python3.8/site-packages/pip_audit/_audit.py", line 66, in audit
    for dep, vulns in self._service.query_all(specs):
  File "/private/var/folders/7b/w1fc1k292kgdj7g01ry8vf_40000gn/T/tmp.TBMsMgV1/.venv/lib/python3.8/site-packages/pip_audit/_service/interface.py", line 148, in query_all
    for spec in specs:
  File "/private/var/folders/7b/w1fc1k292kgdj7g01ry8vf_40000gn/T/tmp.TBMsMgV1/.venv/lib/python3.8/site-packages/pip_audit/_dependency_source/requirement.py", line 120, in collect
    for _, dep in self._collect_cached_deps(filename, reqs):
  File "/private/var/folders/7b/w1fc1k292kgdj7g01ry8vf_40000gn/T/tmp.TBMsMgV1/.venv/lib/python3.8/site-packages/pip_audit/_dependency_source/requirement.py", line 320, in _collect_cached_deps
    for req, resolved_deps in self._resolver.resolve_all(iter(req_values)):
  File "/private/var/folders/7b/w1fc1k292kgdj7g01ry8vf_40000gn/T/tmp.TBMsMgV1/.venv/lib/python3.8/site-packages/pip_audit/_dependency_source/interface.py", line 87, in resolve_all
    yield (req, self.resolve(req))
  File "/private/var/folders/7b/w1fc1k292kgdj7g01ry8vf_40000gn/T/tmp.TBMsMgV1/.venv/lib/python3.8/site-packages/pip_audit/_dependency_source/resolvelib/resolvelib.py", line 75, in resolve
    result = self.resolver.resolve([req])
  File "/private/var/folders/7b/w1fc1k292kgdj7g01ry8vf_40000gn/T/tmp.TBMsMgV1/.venv/lib/python3.8/site-packages/resolvelib/resolvers.py", line 481, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/private/var/folders/7b/w1fc1k292kgdj7g01ry8vf_40000gn/T/tmp.TBMsMgV1/.venv/lib/python3.8/site-packages/resolvelib/resolvers.py", line 373, in resolve
    failure_causes = self._attempt_to_pin_criterion(name)
  File "/private/var/folders/7b/w1fc1k292kgdj7g01ry8vf_40000gn/T/tmp.TBMsMgV1/.venv/lib/python3.8/site-packages/resolvelib/resolvers.py", line 213, in _attempt_to_pin_criterion
    criteria = self._get_updated_criteria(candidate)
  File "/private/var/folders/7b/w1fc1k292kgdj7g01ry8vf_40000gn/T/tmp.TBMsMgV1/.venv/lib/python3.8/site-packages/resolvelib/resolvers.py", line 203, in _get_updated_criteria
    for requirement in self._p.get_dependencies(candidate=candidate):
  File "/private/var/folders/7b/w1fc1k292kgdj7g01ry8vf_40000gn/T/tmp.TBMsMgV1/.venv/lib/python3.8/site-packages/pip_audit/_dependency_source/resolvelib/pypi_provider.py", line 370, in get_dependencies
    return candidate.dependencies
  File "/private/var/folders/7b/w1fc1k292kgdj7g01ry8vf_40000gn/T/tmp.TBMsMgV1/.venv/lib/python3.8/site-packages/pip_audit/_dependency_source/resolvelib/pypi_provider.py", line 120, in dependencies
    self._dependencies = list(self._get_dependencies())
  File "/private/var/folders/7b/w1fc1k292kgdj7g01ry8vf_40000gn/T/tmp.TBMsMgV1/.venv/lib/python3.8/site-packages/pip_audit/_dependency_source/resolvelib/pypi_provider.py", line 102, in _get_dependencies
    deps: List[str] = self.metadata.get_all("Requires-Dist", [])
  File "/private/var/folders/7b/w1fc1k292kgdj7g01ry8vf_40000gn/T/tmp.TBMsMgV1/.venv/lib/python3.8/site-packages/pip_audit/_dependency_source/resolvelib/pypi_provider.py", line 95, in metadata
    self._metadata = self._get_metadata_for_sdist()
  File "/private/var/folders/7b/w1fc1k292kgdj7g01ry8vf_40000gn/T/tmp.TBMsMgV1/.venv/lib/python3.8/site-packages/pip_audit/_dependency_source/resolvelib/pypi_provider.py", line 163, in _get_metadata_for_sdist
    ve.create(ve_dir)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/venv/__init__.py", line 68, in create
    self._setup_pip(context)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/venv/__init__.py", line 289, in _setup_pip
    subprocess.check_output(cmd, stderr=subprocess.STDOUT)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/var/folders/7b/w1fc1k292kgdj7g01ry8vf_40000gn/T/tmpxiqxv9fd/bin/python', '-Im', 'ensurepip', '--upgrade', '--default-pip']' died with <Signals.SIGABRT: 6>.

[woodruffw]

Thanks for the report!

Yes, that's very strange -- I wouldn't have expected ensurepip to ever fail, much less die with a SIGABRT.

Could you fill out the bug report template, in its full detail? In particular, your host and version information would help us get started on root-causing this.

[woodruffw]

This looks potentially related to this CPython bug report: https://bugs.python.org/issue38705

[woodruffw]

Another xref: https://github.com/pypa/build/issues/294

[woodruffw]

Tagged with macOS (intuiting from the paths in the traceback).

[woodruffw]

If I'm right, we basically have two options here:

• We can catch the error and propagate it appropriately, effectively failing the pip-audit because creation of a virtual environment failed. This would be "correct" in terms of how the surrounding host is configured, since there's clearly some kind of layering violation between the host's "system" Python and the one configured in the virtual environment.

• We can attempt to hack around the error here, probably by using venv.EnvBuilder with with_pip=False instead of True, followed by a manual install of pip and setuptools in EnvBuilder.post_setup(). Based on the xrefs above this would probably work and would allow us to continue without an error, but is arguably an end-run around a broken system Python. It also does an end-run around the "correct" way to ensure pip is in the venv, meaning that 99% of users who don't hit this particular bug will now how to rely on the correctness of our approach rather than the Python stdlib.

Thoughts @di?

[tetsuo-cpp]

My bad, I misread this issue. Looks like we want to keep this open until a follow up PR to catch the VirtualEnvError and add some user friendly error message.

[woodruffw]

Yep! Thanks for reopening.

Sent from mobile. Please excuse my brevity.

On Aug 30, 2022, at 3:05 PM, Alex Cameron @.***> wrote:

 My bad, I misread this issue. Looks like we want to keep this open until a follow up PR to catch the VirtualEnvError and add some user friendly error message.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.

[woodruffw]

Re-tagged as upstream to emphasize that we're working around an upstream Python/pip distribution bug here.

[woodruffw]

Doing some triaging: this is purely a macOS Python distribution problem: https://github.com/python/cpython/issues/82886

As such, I'm closing this out: I don't think there's anything actionable that pip-audit itself can do here that wouldn't itself be brittle, unfortunately.