Respect extra-index-url in pip.conf

[matthewdeanmartin]

Is your feature request related to a problem? Please describe.

I'm trying to use pip-audit without ever running setup.py of a package being audited. I have discovered export PIP_ONLY_BINARY=:all: which forces pip to only install if there is a wheel and pip-audit will respect that. So far so good, but now it will blow up on all packages without wheels.

I considered personally convincing every active package on pypi to publish a wheel and then filing a PEP 541 action on the rest, but I'm limited by my finite lifespan.

Fortunately, piwheels has republished everything on pypi in wheel format and accidentally solved this * for me. So I edit ~/pip/pip.conf and add extra-index-url=https://www.piwheels.org/simple

requirements.txt has only termcolor==*

$ export PIP_AUDIT_LOGLEVEL=debug
$ export PIP_ONLY_BINARY=:all:
$ pip-audit -r requirements.txt --verbose
DEBUG:pip_audit._cli:parsed arguments: Namespace(local=False, requirements=[<_io.TextIOWrapper name='requirements.txt' mode='r' encoding='cp1252'>], format=<OutputFormatChoice.Columns: 'columns'>, vulnerability_service=<VulnerabilityServiceChoice.Pypi: 'pypi'>, dry_run=False, strict=False, desc=<VulnerabilityDescriptionChoice.Auto: 'auto'>, cache_dir=None, progress_spinner=<ProgressSpinnerChoice.On: 'on'>, timeout=15, paths=[], verbose=True)
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): pypi.org:443
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/termcolor HTTP/1.1" 301 215
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/termcolor/ HTTP/1.1" 200 875
Traceback (most recent call last):
  File "C:\Users\matth\.virtualenvs\pip-audit-GpdA97f3\lib\site-packages\resolvelib\resolvers.py", line 348, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "C:\Users\matth\.virtualenvs\pip-audit-GpdA97f3\lib\site-packages\resolvelib\resolvers.py", line 173, in _add_to_criteria
    raise RequirementsConflicted(criterion)
resolvelib.resolvers.RequirementsConflicted: Requirements conflict: <Requirement('termcolor==*')>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\matth\.virtualenvs\pip-audit-GpdA97f3\Scripts\pip-audit-script.py", line 33, in <module>
    sys.exit(load_entry_point('pip-audit', 'console_scripts', 'pip-audit')())
  File "c:\users\matth\pip-audit\pip_audit\_cli.py", line 263, in audit
    for (spec, vulns) in auditor.audit(source):
  File "c:\users\matth\pip-audit\pip_audit\_audit.py", line 60, in audit
    yield from self._service.query_all(specs)
  File "c:\users\matth\pip-audit\pip_audit\_service\interface.py", line 115, in query_all
    for spec in specs:
  File "c:\users\matth\pip-audit\pip_audit\_dependency_source\requirement.py", line 63, in collect
    for _, deps in self.resolver.resolve_all(iter(req_values)):
  File "c:\users\matth\pip-audit\pip_audit\_dependency_source\interface.py", line 67, in resolve_all
    yield (req, self.resolve(req))
  File "c:\users\matth\pip-audit\pip_audit\_dependency_source\resolvelib\resolvelib.py", line 44, in resolve
    result = self.resolver.resolve([req])
  File "C:\Users\matth\.virtualenvs\pip-audit-GpdA97f3\lib\site-packages\resolvelib\resolvers.py", line 481, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "C:\Users\matth\.virtualenvs\pip-audit-GpdA97f3\lib\site-packages\resolvelib\resolvers.py", line 350, in resolve
    raise ResolutionImpossible(e.criterion.information)
resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=<Requirement('termcolor==*')>, parent=None)]

matth@DESKTOP-FANFFGU MINGW64 ~/pip-audit (main)
$ pip install --verbose termcolor
Using pip 21.3.1 from C:\Users\matth\.virtualenvs\pip-audit-GpdA97f3\lib\site-packages\pip (python 3.9)
Looking in indexes: https://pypi.org/simple, http://pypi.org/simple, https://www.piwheels.org/simple
Collecting termcolor
  Downloading https://www.piwheels.org/simple/termcolor/termcolor-1.1.0-py3-none-any.whl (4.8 kB)
Installing collected packages: termcolor
Successfully installed termcolor-1.1.0

That demonstrates that pip-audit respect the 'install only wheels' flag, but doesn't respect the 'download wheels from the only place that has them all' flag, but pip does.

Describe the solution you'd like Do what pip does.

Describe alternatives you've considered PEP 541 against all packages without wheels.

Additional context N/A

[di]

Hi @matthewdeanmartin, thanks for the issue! I think this is probably a duplicate of https://github.com/trailofbits/pip-audit/issues/46, but I'll let you decide.

I considered personally convincing every active package on pypi to publish a wheel and then filing a PEP 541 action on the rest, but I'm limited by my finite lifespan.

You and me both, buddy 🙃

[woodruffw]

I was also going to xref #46, but I think we can leave this open as a subissue, since it specifically concerns one possible way we can support additional indices (respecting the user's pip.conf, rather than requiring the user to duplicate all of their index information for both pip and pip-audit).

[matthewdeanmartin]

@woodruffw you read my mind. Getting my devs to config pip.conf correctly is challenge enough without teaching them two ways to list the pypi servers and their precedence.

I see the challenge of deciding if package from private server X is problematic but the same package on public pypi is not & vica versa (i.e. guess the names of packages that corps use on their private pypi repos, e.g. apple_corp_python_code on a private server is good, but on a pypi server it is probably an attempt to hack apple corp)

[woodruffw]

CC @jleightcap: this is another good one to look at!