importlib_metadata.PackageNotFoundError: jeepney

[carlosdoordash]

We run pipenv update --dev in our CI pipeline for a while now. After pipenv 2022.9.8 bumped its version of pipdeptree #5343 , we are now getting the following stacktrace which seems related to your recent change.

If I were to guess, it looks like you are catching ImportError but not importlib_metadata.PackageNotFoundError.

[2022-09-08T16:20:55.706Z] pipenv update --dev
[2022-09-08T16:20:55.961Z] Running $ pipenv lock then $ pipenv sync.
[2022-09-08T16:20:55.962Z] Locking [packages] dependencies...
[2022-09-08T16:20:55.962Z] Locking [dev-packages] dependencies...
[2022-09-08T16:20:55.962Z] 
Building requirements...
[2022-09-08T16:20:55.962Z] 
Resolving dependencies...
[2022-09-08T16:21:01.205Z] 
FAIL
[2022-09-08T16:21:01.205Z] 
[2022-09-08T16:21:01.205Z] Traceback (most recent call last):
[2022-09-08T16:21:01.205Z]   File "/usr/local/lib/python3.7/site-packages/pipenv/resolver.py", line 867, in <module>
[2022-09-08T16:21:01.205Z]     main()
[2022-09-08T16:21:01.205Z]   File "/usr/local/lib/python3.7/site-packages/pipenv/resolver.py", line 862, in main
[2022-09-08T16:21:01.205Z]     dev=parsed.dev,
[2022-09-08T16:21:01.205Z]   File "/usr/local/lib/python3.7/site-packages/pipenv/resolver.py", line 834, in _main
[2022-09-08T16:21:01.205Z]     pre, clear, verbose, system, write, requirements_dir, packages, dev
[2022-09-08T16:21:01.205Z]   File "/usr/local/lib/python3.7/site-packages/pipenv/resolver.py", line 794, in resolve_packages
[2022-09-08T16:21:01.205Z]     results = clean_results(results, resolver, project, dev)
[2022-09-08T16:21:01.205Z]   File "/usr/local/lib/python3.7/site-packages/pipenv/resolver.py", line 651, in clean_results
[2022-09-08T16:21:01.205Z]     reverse_deps = project.environment.reverse_dependencies()
[2022-09-08T16:21:01.205Z]   File "/usr/local/lib/python3.7/site-packages/pipenv/environment.py", line 730, in reverse_dependencies
[2022-09-08T16:21:01.205Z]     for req in self.get_package_requirements():
[2022-09-08T16:21:01.205Z]   File "/usr/local/lib/python3.7/site-packages/pipenv/environment.py", line 710, in get_package_requirements
[2022-09-08T16:21:01.205Z]     return [self._get_requirements_for_package(p, key_tree) for p in nodes]
[2022-09-08T16:21:01.205Z]   File "/usr/local/lib/python3.7/site-packages/pipenv/environment.py", line 710, in <listcomp>
[2022-09-08T16:21:01.205Z]     return [self._get_requirements_for_package(p, key_tree) for p in nodes]
[2022-09-08T16:21:01.205Z]   File "/usr/local/lib/python3.7/site-packages/pipenv/environment.py", line 685, in _get_requirements_for_package
[2022-09-08T16:21:01.205Z]     for c in get_children(node)
[2022-09-08T16:21:01.205Z]   File "/usr/local/lib/python3.7/site-packages/pipenv/environment.py", line 686, in <listcomp>
[2022-09-08T16:21:01.205Z]     if c.project_name not in chain
[2022-09-08T16:21:01.205Z]   File "/usr/local/lib/python3.7/site-packages/pipenv/environment.py", line 685, in _get_requirements_for_package
[2022-09-08T16:21:01.205Z]     for c in get_children(node)
[2022-09-08T16:21:01.205Z]   File "/usr/local/lib/python3.7/site-packages/pipenv/environment.py", line 686, in <listcomp>
[2022-09-08T16:21:01.205Z]     if c.project_name not in chain
[2022-09-08T16:21:01.205Z]   File "/usr/local/lib/python3.7/site-packages/pipenv/environment.py", line 673, in _get_requirements_for_package
[2022-09-08T16:21:01.205Z]     d = node.as_dict()
[2022-09-08T16:21:01.206Z]   File "/usr/local/lib/python3.7/site-packages/pipenv/vendor/pipdeptree/__init__.py", line 241, in as_dict
[2022-09-08T16:21:01.206Z]     "installed_version": self.installed_version,
[2022-09-08T16:21:01.206Z]   File "/usr/local/lib/python3.7/site-packages/pipenv/vendor/pipdeptree/__init__.py", line 205, in installed_version
[2022-09-08T16:21:01.206Z]     return guess_version(self.key, self.UNKNOWN_VERSION)
[2022-09-08T16:21:01.206Z]   File "/usr/local/lib/python3.7/site-packages/pipenv/vendor/pipdeptree/__init__.py", line 49, in guess_version
[2022-09-08T16:21:01.206Z]     return importlib_metadata.version(pkg_key)
[2022-09-08T16:21:01.206Z]   File "/root/.local/share/virtualenvs/cd.yml-X1meJv26/lib/python3.7/site-packages/importlib_metadata/__init__.py", line 553, in version
[2022-09-08T16:21:01.206Z]     return distribution(distribution_name).version
[2022-09-08T16:21:01.206Z]   File "/root/.local/share/virtualenvs/cd.yml-X1meJv26/lib/python3.7/site-packages/importlib_metadata/__init__.py", line 526, in distribution
[2022-09-08T16:21:01.206Z]     return Distribution.from_name(distribution_name)
[2022-09-08T16:21:01.206Z]   File "/root/.local/share/virtualenvs/cd.yml-X1meJv26/lib/python3.7/site-packages/importlib_metadata/__init__.py", line 194, in from_name
[2022-09-08T16:21:01.206Z]     raise PackageNotFoundError(name)
[2022-09-08T16:21:01.206Z] importlib_metadata.PackageNotFoundError: jeepney

[saaketp]

I actually did consider catching that separately, but skipped it because PackageNotFoundError is already a subclass of ImportError https://github.com/python/importlib_metadata/blob/53e42d6d70bd2068cd5bf0d351df89aa3cc0af31/importlib_metadata/init.py#L51

But looks like it wasn't a subclass of ImportError the past, it was changed 3 years ago in version 0.13 https://github.com/python/importlib_metadata/commit/73651fd9b8df0c9c79de087b61246294bf89418f And if we go even farther back in time then PackageNotFoundError didn't even exist https://github.com/python/importlib_metadata/commit/12a1e5da835d5552c113497c465553cd8df57858.

So maybe the fix should be to except Exception to handle all problems with importlib_metadata API changes.

You can also fix this on your end by using a newer version of importlib_metadata.

[saaketp]

Actually I was wrong about the 3 years ago thing, the BaseClass is either ImportError or ModuleNotFoundError (which is subclass of ImportError) even in the original version that introduced PackageNotFoundError https://github.com/python/importlib_metadata/blob/12a1e5da835d5552c113497c465553cd8df57858/importlib_metadata/api.py#L7-L10.

So not sure why you get the error, and probably updating importlib_metadata won't help you.

[carlosdoordash]

I see your point and it is backed up by https://docs.python.org/3.7/library/exceptions.html#ImportError

In the meantime, we are pinned to pipenv==2022.9.4 which is using pipdeptree==2.2.1

[kemzeb]

After comparing your stack trace to importlib_metadata source code, I believe you may have been running importlib_metadata v1.5.0 or v1.5.1 at the time since the line numbers in the stack trace match exactly to the source code of v1.5.0/v1.5.1 (assuming the distribution package doesn't optimize away some of the code like a compiler would do; from looking at the source of some of my installed packages it doesn't look like it does). Every other version I could find did not have them align in such a way.

From looking at its code, the problem appears to be here. The builtin ModuleNotFoundError is not used, but rather importlib_metadata creates its own version and assigns it to builtins.FileNotFoundError.

Not sure why the code has been written this way (FileNotFoundError has been around since Python v3.3, but ModuleNotFound was introduced in Python v3.6), but this error is not a subclass of ImportError, which is why I believe we weren't able to catch it. It looks like v3.0.0 onwards it does use the the builtin ModuleNotFoundError.

Going to close this as I believe this is the problem and on top of this we done away with using ImportError and instead use PackageNotFoundError directly. Please let me know if have any further questions @carlosdoordash!