Having private packages makes python-inspector crashes

[qequ]

This issue is originally from ort, that uses python-inspector. When having a private package in the poetry.lock makes python-inspector crashes with the following report

  File "/home/alvaro/ort2/env_ort/lib/python3.10/site-packages/python_inspector/resolution.py", line 581, in _iter_matches
    raise NoVersionsFound(f"This package does not exist: {name}")
python_inspector.error.NoVersionsFound: This package does not exist: private-package

[pombredanne]

@qequ Thanks for the report! Do you have a reproducible example to share?

[qequ]

I'm not using python-inspector directly but through ort with a python project using poetry. ort calls it with a requirements.txt that looks like this

--extra-index-url https://gitlab.com/api/v4/.../.../.../../...

anyio==3.6.1 ; python_version >= "3.10" and python_version < "4.0"
boto3==1.24.16 ; python_version >= "3.10" and python_version < "4.0"
botocore==1.27.16 ; python_version >= "3.10" and python_version < "4.0"
click==8.1.3 ; python_version >= "3.10" and python_version < "4.0"
dateparser==1.1.1 ; python_version >= "3.10" and python_version < "4.0"
fastapi==0.78.0 ; python_version >= "3.10" and python_version < "4.0"
idna==3.3 ; python_version >= "3.10" and python_version < "4.0"
pydantic==1.10.2 ; python_version >= "3.10" and python_version < "4.0"
pyparsing==3.0.9 ; python_version >= "3.10" and python_version < "4.0"
python-dateutil==2.8.2 ; python_version >= "3.10" and python_version < "4.0"
pytz-deprecation-shim==0.1.0.post0 ; python_version >= "3.10" and python_version < "4.0"
pytz==2022.1 ; python_version >= "3.10" and python_version < "4.0"
s3transfer==0.6.0 ; python_version >= "3.10" and python_version < "4.0"
six==1.16.0 ; python_version >= "3.10" and python_version < "4.0"
sniffio==1.2.0 ; python_version >= "3.10" and python_version < "4.0"
starlette==0.19.1 ; python_version >= "3.10" and python_version < "4.0"
private-package==1.0.2 ; python_version >= "3.10" and python_version < "4.0"
typing-extensions==4.2.0 ; python_version >= "3.10" and python_version < "4.0"
tzdata==2022.1 ; python_version >= "3.10" and python_version < "4.0"
tzlocal==4.2 ; python_version >= "3.10" and python_version < "4.0"
urllib3==1.26.9 ; python_version >= "3.10" and python_version < "4"
uvicorn==0.18.1 ; python_version >= "3.10" and python_version < "4.0"

where private-package is a package hosted privately in a gitlab repo set in the --index-url at the top. python-inspector founding the private packge

[JukMR]

    raise NoVersionsFound(f"This package does not exist: {name}")
python_inspector.error.NoVersionsFound: This package does not exist: repo2

I've run into the same issue. I will add my example to replicate this.

In the following repo: https://github.com/JukMR/oss_demo.git

Trying to run:

➜  repo1 git:(main) ✗ python-inspector --python-version 310 --operating-system linux --json-pdt this.json --analyze-setup-py-insecurely --requirement /home/julianmr/oss_demo/repo1/requirements.txt --verbose

yields the following output:

➜  repo1 git:(main) ✗ python-inspector --python-version 310 --operating-system linux --json-pdt this.json --analyze-setup-py-insecurely --requirement /home/julianmr/oss_demo/repo1/requirements.txt --verbose
Resolving dependencies...
Using netrc file /home/julianmr/.netrc
direct_dependencies:
 DependentPackage(purl='pkg:pypi/[email protected]', extracted_requirement='repo2==1.0.0', scope='install')
environment: Environment(python_version='310', operating_system='linux')
repos:
 PypiSimpleRepository(index_url='https://pypi.org/simple', credentials=None)
Traceback (most recent call last):
  File "/home/julianmr/.local/lib/python3.10/site-packages/python_inspector/resolve_cli.py", line 247, in resolve_dependencies
    resolution_result: Dict = resolver_api(
  File "/home/julianmr/.local/lib/python3.10/site-packages/python_inspector/api.py", line 263, in resolve_dependencies
    resolution, purls = resolve(
  File "/home/julianmr/.local/lib/python3.10/site-packages/python_inspector/api.py", line 322, in resolve
    resolved_dependencies, packages = get_resolved_dependencies(
  File "/home/julianmr/.local/lib/python3.10/site-packages/python_inspector/api.py", line 360, in get_resolved_dependencies
    resolver_results = resolver.resolve(requirements=requirements, max_rounds=max_rounds)
  File "/home/julianmr/.local/lib/python3.10/site-packages/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/home/julianmr/.local/lib/python3.10/site-packages/resolvelib/resolvers.py", line 397, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "/home/julianmr/.local/lib/python3.10/site-packages/resolvelib/resolvers.py", line 148, in _add_to_criteria
    matches = self._p.find_matches(
  File "/home/julianmr/.local/lib/python3.10/site-packages/python_inspector/resolution.py", line 604, in find_matches
    candidates = sorted(
  File "/home/julianmr/.local/lib/python3.10/site-packages/python_inspector/resolution.py", line 586, in _iter_matches
    raise NoVersionsFound(f"This package does not exist: {name}")
python_inspector.error.NoVersionsFound: This package does not exist: repo2

The problem here is that repo2 is a local package built by me and installed using pip install -e . which is not available in PyPi but can be seen installed locally by running:

➜  oss_demo git:(main) ✗ pip list | grep repo
repo1                        0.0.0        /home/julianmr/oss_demo/repo1
repo2                        1.0.0        /home/julianmr/oss_demo/repo2
repo3                        1.0.0        /home/julianmr/oss_demo/repo3

[pombredanne]

@JukMR python-inspector does not know anything about local private packages installed in the current interpreter. All packages whether public or private need to be somehow accessible in some repository that's been setup. In your example, if I checkout your repo and cd to repo1 and run pip this fails too:

oss_demo/repo1$ pip install -r requirements.txt 
ERROR: Could not find a version that satisfies the requirement repo2==1.0.0 (from versions: none)
ERROR: No matching distribution found for repo2==1.0.0

[pombredanne]

@qequ just to be clear... you wrote above in https://github.com/nexB/python-inspector/issues/127#issuecomment-1473871521 :

where private-package is a package hosted privately in a gitlab repo set in the --index-url at the top. python-inspector founding the private packge

where emphasis is mine.... I understood back then that python-inspector found your private package correctly after all, right?

[dgutson]

@luciamartinezgavier

[qequ]

@pombredanne no, I ran into the same issue that @JukMR had. It crashed when failling to find the private package