pip Add additional diagnostics to our error messages

What's the problem this feature will solve?

pip's error messages do not provide any context on what component or thing is involved in the problem, and where the bug/issue might lie.

Describe the solution you'd like

Add clear diagnostic information about the $thing that likely caused the error (eg: package, network, index page). We could also add hints toward potential solutions, in cases where we know the likely causes (eg: this is an issue with <url>, or this package is not supported on your platform, or did you setup an environment to build the package).

Alternative Solutions

Maintain status quo? We've got adhoc messages right now, which may or may not contain diagnostic information for the user, usually not.

Additional context

This is a pattern of behaviour I've seen on the issue tracker, in conversational feedback about pip's error messages, and seems to align with my personal effort of "what can we learn from other package managers to improve".

Sep 02 '21 08:09 pradyunsg

+1 from me, I guess. Presumably the intent is to keep this as an umbrella issue, kept open indefinitely, with individual PRs linked to it as we change messages? Would a project be better for this? Or are you thinking of this as being about implementing a more structured "generate an error" API, a bit like the deprecation helper?

I imagine there will be some low-hanging fruit here along with some much trickier ones (ahem resolver).

Sep 02 '21 09:09 pfmoore

Well, mostly to gain consensus before I try tackling this in a more structured manner; similar to the documentation rewrite. I'll likely end up doing the same thing -- keep the issue open and backlink to this; with a project for tracking progress.

Sep 02 '21 12:09 pradyunsg

Cool, I see no problem with doing this, it seems like it can only improve the UX when things go wrong.

Sep 02 '21 12:09 pfmoore

I'm also +1 on this change.

Doing this can also reduce the number of issues based on obscure error messages. WIth clear messages, the user may understand what to do, instead of opening a new GitHub issue :)

Sep 02 '21 17:09 DiddiLeija

If anyone has suggestions for places to improve or wants to cross-reference existing issues to this one, I’m all ears for that. The more potential enhancements we can find that fit this, the better I’ll be able to figure out what to spend time on! :)

Sep 02 '21 17:09 pradyunsg

https://github.com/pypa/pip/issues/8262#issuecomment-910762685 is an example. Explaining that the error is from the package rather than pip would help.

I also wonder if tweaking pip-crashed tracebacks to be more distinct from regular Python tracebacks would help in these situations.

Sep 02 '21 17:09 pradyunsg

https://github.com/pypa/pip/issues/6584 as well!

Sep 03 '21 09:09 pradyunsg

https://github.com/pypa/pip/pull/8738 is another.

Sep 18 '21 08:09 pradyunsg

Looking at https://github.com/pypa/pip/issues/8618, it looks like the failure to build packages trips a lot of new users as well!

Sep 19 '21 13:09 pradyunsg

Alrighty. So, here's the things that I've spent basically all day thinking through today:

the presentation style for errors-with-diagnostics.
how this overlaps with our past ideas about how we present output / errors.
an approach to go about implementing this in our codebase, incrementally.

I'll try to go through and summarise my thoughts on each of these one-by-one. This is definitely open to all kinds of feedback and inputs; and I'm likely gonna throw out a tweet or two to get the broader community to wieght in on this as well.

Here's the presentation style I've settled on:

(output streams that will be happy with unicode)

{kind}: {reference}

{message}
╰─▶ {cause}

⚠️ {attention message}
💡 Hint: {hint message}

(all the other "lame" streams)

{kind}: {reference}

{message}
Caused by: {cause}

Note: {attention message}
Hint: {hint message}

Note that the {reference} ids in this can be made into hyperlinks as well, when supported -- https://gist.github.com/egmontkob/eb114294efbcd5adb1944c9f3cb5feda -- and it would be pretty neat to link to our documentation from this. :)

As an example, here's what we've go today:

❯ pip install reqeusts
ERROR: Could not find a version that satisfies the requirement reqeusts (from versions: none)
ERROR: No matching distribution found for reqeusts

Here's how that'll look with this model:

$ pip install reqeusts
Collecting reqeusts
Error: no-available-distribution-files

× Could not find any distribution files for reqeusts
╰─▶ 0 distribution files were found

💡 Hint: Is there a typo in the package name?

That hint at the end is triggered by "Is number of available distributions for this zero?", which... well... it's quirky to get that to be shown only in the right situations. :(

Here's another example, demonstrating the issue with that:

Collecting xdict
  Could not fetch URL https://pypi.python.org/simple/xdict/: There was a problem confirming the ssl certificate: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:590) - skipping
  Could not find a version that satisfies the requirement xdict (from versions: none)
No matching distribution found for xdict

With the diagnostics model I'm suggesting here, we'd present the same information in the following format instead:

Collecting xdict
  Warning: network-connectivity-ssl

  × Skipped URL: https://pypi.python.org/simple/xdict/
  ╰─▶ There was a problem with the SSL connection!
      [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:590)

  ⚠️ This is likely an issue with the network, or pip's network configuration.

Error: no-available-distribution-files

× Could not find any distribution files for xdict
╰─▶ 0 distribution files were found

💡 Hint: Is there a typo in the package name?

If we can manage adding some complexity, for passing the information about the "get-page" failure, to the point in the code where we actually raise the "No matching distributions" error -- we can do something like:

Collecting xdict
  Warning: network-connectivity (SSL)

  × Skipped URL: https://pypi.python.org/simple/xdict/
  ╰─▶ There was a problem with the SSL connection!
      [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:590)

Error: no-available-distribution-files

× Could not find any distribution files for xdict
╰─▶ 0 distribution files were found

⚠️ This was likely caused by an problem with the network connectivity, or pip's network configuration.
💡 Hint: The details of the network issue was printed by pip, when it occurred.

And, perhaps most importantly, I still need to figure out how to get pretty colours working in this model. This might need something drastic like #10423 to be done though. 🤷🏽

This also makes progress on a few blanket issues/broader ideas we have:

#6119: That's what the {reference} becomes. Examples I've got so far are things like reported-metadata-file-missing, metadata-generation-failed and network-connectivity-ssl.
#6541: We'd create a consistent style for our error reporting, which means we'd effectively have something like our deprecated helper -- but for error messages. Similar to that helper, the hope is that we'd be able to move over all of our error reporting to this mechanism incrementally; until it's all moved over and then we've basically solved a substantial chunk of that issue. :)

This also interacts well with:

#9475: The references can effectively be the URL slugs for headings under an overall "errors" index, that I am hoping to be able to create at some point in the future (as part of #9475) -- taking inspiration from https://doc.rust-lang.org/error-index.html.
#4649: This drastically improves error messages and provides easier-to-spot guidance IMO. That was part of the motivation toward that issue (the other part there is our long-scrollback-output behaviour, which this won't solve).
#6526: There's a clearer mechanism to provide additional context and information.
#5380: There's a clearer mechanism to provide additional context and information.

I'm still thinking about whether this should live as code within pip, or be a separate dependency that we vendor. I'm leaning toward the former, since I don't want to take on the maintainance overhead of providing such a package more broadly (OTOH, it'd be REALLY NICE!). Anyway, it is definitely possible to implement things in a way that keeps our options open, and I intend to do that.

The idea is essentially to introduce a DiagnosticPipError that we'd catch at our top-level exception handler. The main presentation logic will be held in that exception. All derived exceptions will take in the appropriate content from the point-of-error, and call super().__init__ with the properly formed chunks with appropriate diagnostics.

This roughly translates to looking with the following shape:

class DiagnosticPipError(PipError):
    reference: str
    message: str
    kind: Literal["error", "warning"]
    reference: Optional[str]
    attention: Optional[str]
    hint: Optional[str]


class MetadataGenerationFailure(DiagnosticPipError):
    reference = "metadata-generation-failed"

    def __init__(self, failure: Any) -> None:
        # Do checks on the failure to figure out the right values here.
        super().__init__(
            message=...,
            context=...,
            attention="This is likely an issue with how the package has to be built/installed.",
        )


try:
   ...
except DiagnosticPipError as exc:
   present_diagnostic(exc)

The things that we'd need to do, in no particular order, to adopt this model more completely would be:

Introduce more fine-grained exceptions that derive from DiagnosticPipError.
Convert all our existing exceptions into sub-classes of DiagnosticPipError -- moving existing presnetation logic into those classes in the process.
Updating existing logic to add additional context to the diagnostic error messages -- in the form of better cause information, hints and "attention" notes to guide users in the right direction for their failures.

All of this can be done incrementally, and that's a good thing. :)

Moving the handling of the "what do I present" logic into the exceptions, provides a clear place for that code and also creates a clearer boundary between the "core logic" vs "presentation logic". And, as we adopt this more-and-more, we'd also be creating a bigger gap between pip-crashed vs pip-presented-an-error -- which is also a good thing. :)

Oct 01 '21 21:10 pradyunsg

...
  Downloading [redacted]/78/62/9e38f9b22efe08ec2b40a56c0f46848ce03c35fdd6e78ae445589f914462/cryptography-2.1.2.tar.gz (441 kB)
    ERROR: Command errored out with exit status 1:
     command: /opt/bb/bin/python3.10 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-gklwy3yo/cryptography_9b17be675eec428baaa58a827cac0601/setup.py'"'"'; __file__='"'"'/tmp/pip-install-gklwy3yo/cryptography_9b17be675eec428baaa58a827cac0601/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-jd7k2mxe
         cwd: /tmp/pip-install-gklwy3yo/cryptography_9b17be675eec428baaa58a827cac0601/
    Complete output (3 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ModuleNotFoundError: No module named 'setuptools'
    ----------------------------------------
WARNING: Discarding [redacted]/78/62/9e38f9b22efe08ec2b40a56c0f46848ce03c35fdd6e78ae445589f914462/cryptography-2.1.2.tar.gz#sha256=d7f348e4f5df146a0e75998544bab6d42313cf19a81a6e49990ab7b27cc9c73b (from [redacted]/simple/cryptography/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

We can do better than that, when there's no setuptools installed in the global environment and building an sdist via the legacy build system interface fails.

Oct 06 '21 13:10 pradyunsg

I am quite impressed that it was merged, i really liked the PR!

Oct 29 '21 16:10 Abdur-rahmaanJ

Screenshot 2021-11-12 at 17 06 27

Uh... so... here's a teaser. :)

Nov 12 '21 17:11 pradyunsg

Current state of affairs here is:

#10423 -- we're fine with adopting rich for making this (and other things) easier.
- #10461 is a tracking issue for that.
- #10462 is a PR to do that.
  - https://github.com/pygments/pygments/issues/1953 is an issue to check that pygments is OK being vendored.
  - rich's author as confirmed that he's OK with being vendored in pip.
I've been experimenting with nice error messages in https://github.com/pradyunsg/sphinx-theme-builder, and will be utilizing a lot of that experience here.
I've prototyped/implemented a few improved error messages following this model, mainly around the build system and subprocesses (see teaser above), that change the presentation style of these errors to follow the format described earlier in this thread.
- The plumbing to get them working with our current logging setup is... it's... it works.
- I'll file PRs for this once #10462 lands.
- There's a decent amount of complexity here, and I'd like to be not file too many PRs at the same time.

Nov 12 '21 17:11 pradyunsg

As mentioned, I learnt a bunch of things about error message presentation while working on https://github.com/pradyunsg/sphinx-theme-builder. Transferring some of that presentation and implementation insights into this codebase in #10703.

Dec 05 '21 12:12 pradyunsg

Another update:

#10763 (which is from a first time contributor!) improves how our resolver error messages are presented.
#10795 is a mature version of the prototype mentioned above!

Jan 15 '22 08:01 pradyunsg

Another situation for this -- poorly formatted dependencies on a package during development results in a horrible looking exception.

❯ pip install -e .   
Obtaining file:///Users/pradyunsg/Developer/github/sphinx-theme-builder
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
ERROR: Exception:
Traceback (most recent call last):
  File "/Users/pradyunsg/Developer/github/sphinx-theme-builder/.venv/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3021, in _dep_map
    return self.__dep_map
  File "/Users/pradyunsg/Developer/github/sphinx-theme-builder/.venv/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2815, in __getattr__
    raise AttributeError(attr)
AttributeError: _DistInfoDistribution__dep_map

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/pradyunsg/Developer/github/sphinx-theme-builder/.venv/lib/python3.9/site-packages/pip/_vendor/packaging/requirements.py", line 102, in __init__
    req = REQUIREMENT.parseString(requirement_string)
  File "/Users/pradyunsg/Developer/github/sphinx-theme-builder/.venv/lib/python3.9/site-packages/pip/_vendor/pyparsing/core.py", line 1143, in parse_string
    raise exc.with_traceback(None)
pip._vendor.pyparsing.exceptions.ParseException: Expected string_end, found ';'  (at char 18), (line:1, col:19)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/pradyunsg/Developer/github/sphinx-theme-builder/.venv/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3101, in __init__
    super(Requirement, self).__init__(requirement_string)
  File "/Users/pradyunsg/Developer/github/sphinx-theme-builder/.venv/lib/python3.9/site-packages/pip/_vendor/packaging/requirements.py", line 104, in __init__
    raise InvalidRequirement(
pip._vendor.packaging.requirements.InvalidRequirement: Parse error at "'; python'": Expected string_end

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/pradyunsg/Developer/github/sphinx-theme-builder/.venv/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
    status = run_func(*args)
  File "/Users/pradyunsg/Developer/github/sphinx-theme-builder/.venv/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
    return func(self, options, args)
  File "/Users/pradyunsg/Developer/github/sphinx-theme-builder/.venv/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 341, in run
    requirement_set = resolver.resolve(
  File "/Users/pradyunsg/Developer/github/sphinx-theme-builder/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 94, in resolve
    result = self._result = resolver.resolve(
  File "/Users/pradyunsg/Developer/github/sphinx-theme-builder/.venv/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 481, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/Users/pradyunsg/Developer/github/sphinx-theme-builder/.venv/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 373, in resolve
    failure_causes = self._attempt_to_pin_criterion(name)
  File "/Users/pradyunsg/Developer/github/sphinx-theme-builder/.venv/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 213, in _attempt_to_pin_criterion
    criteria = self._get_updated_criteria(candidate)
  File "/Users/pradyunsg/Developer/github/sphinx-theme-builder/.venv/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 203, in _get_updated_criteria
    for requirement in self._p.get_dependencies(candidate=candidate):
  File "/Users/pradyunsg/Developer/github/sphinx-theme-builder/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/provider.py", line 237, in get_dependencies
    return [r for r in candidate.iter_dependencies(with_requires) if r is not None]
  File "/Users/pradyunsg/Developer/github/sphinx-theme-builder/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/provider.py", line 237, in <listcomp>
    return [r for r in candidate.iter_dependencies(with_requires) if r is not None]
  File "/Users/pradyunsg/Developer/github/sphinx-theme-builder/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 246, in iter_dependencies
    requires = self.dist.iter_dependencies() if with_requires else ()
  File "/Users/pradyunsg/Developer/github/sphinx-theme-builder/.venv/lib/python3.9/site-packages/pip/_internal/metadata/pkg_resources.py", line 200, in iter_dependencies
    return self._dist.requires(extras)
  File "/Users/pradyunsg/Developer/github/sphinx-theme-builder/.venv/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2736, in requires
    dm = self._dep_map
  File "/Users/pradyunsg/Developer/github/sphinx-theme-builder/.venv/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3023, in _dep_map
    self.__dep_map = self._compute_dependencies()
  File "/Users/pradyunsg/Developer/github/sphinx-theme-builder/.venv/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3033, in _compute_dependencies
    reqs.extend(parse_requirements(req))
  File "/Users/pradyunsg/Developer/github/sphinx-theme-builder/.venv/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3094, in parse_requirements
    yield Requirement(line)
  File "/Users/pradyunsg/Developer/github/sphinx-theme-builder/.venv/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3103, in __init__
    raise RequirementParseError(str(e))
pip._vendor.pkg_resources.RequirementParseError: Parse error at "'; python'": Expected string_end

Jun 04 '22 20:06 pradyunsg

PreviousBuildDirError is a good candidate as well, since it's already fairly special-cased.

Jun 04 '22 20:06 pradyunsg

As outlined in https://github.com/pypa/pip/issues/11816 - a trace message should not be thrown without explanation what the circumstances are. E.g. the version number of the tools pip and python, the environment being detected OS/venv and key settings like user or global install should be shown with general hints where to find help for the problem e.g. home page of the tools issue list and docs. It's also o.k. to do this in full detail only if a certain flag/environment variable is set and point this out in a shorter message which is the standard.

Feb 27 '23 06:02 WolfgangFahl

I'm not sure about the current state of the issue.

Apologies to anyone that's not interested 🙃

I'll mention my recent experience with docker run -it --rm --entrypoint ash python:3.11-alpine, pip install --only-binary ":all:" 'pandas~=2.0.3' -vv and pandas~=2.0.3:

Now there's:

Fetched page https://pypi.org/simple/pandas/ as application/vnd.pypi.simple.v1+json
  Found link https://files.pythonhosted.org/packages/bc/ad/d1f0a867064f62ffde917876cc09cfd53352af2b1f147c140fd1943a0c7a/pandas-2.1.0-cp311-cp311-musllinux_1_1_x86_64.whl (from https://pypi.org/simple/pandas/) (requires-python:>=3.9), version: 2.1.0
  Found link https://files.pythonhosted.org/packages/db/b0/f59da3679161250e8d5414e2a8b11892015e9f7bdcb10e53d2792c6b7b51/pandas-2.1.0rc0-cp311-cp311-musllinux_1_1_x86_64.whl (from https://pypi.org/simple/pandas/) (requires-python:>=3.9), version: 2.1.0rc0
  Skipping link: none of the wheel's tags (cp310-cp310-manylinux2014_aarch64, cp310-cp310-manylinux_2_17_aarch64) are compatible (run pip debug --verbose to show compatible tags): https://files.pythonhosted.org/packages/10/df/3a4b53426bb62b46c4744712f8f7443ec788bd04d167599d55c0244e1c10/pandas-2.0.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (from https://pypi.org/simple/pandas/) (requires-python:>=3.8)
...
  Skipping link: none of the wheel's tags (cp39-cp39-musllinux_1_1_x86_64) are compatible (run pip debug --verbose to show compatible tags): https://files.pythonhosted.org/packages/ca/14/00cf15abbb69367d35226216bb6e29804fb7e21687b35a8de474a288ffb0/pandas-2.1.0rc0-cp39-cp39-musllinux_1_1_x86_64.whl (from https://pypi.org/simple/pandas/) (requires-python:>=3.9)
Skipping link: not a file: https://pypi.org/simple/pandas/
Given no hashes to check 0 links for project 'pandas': discarding no candidates

That looks amazing 😍

Now, just a fewww improvements on that, and it'd be perfect (and hopefully not require -vv): Sort by issue:

Probably, I'd want my options to fit my specifier (~=2.0.3). I don't care if pip found a suitable pandas-0
Then, I'd want my options to fit my OS. I don't care if pip found Windows or Mac OS compatible ones
Then, I'd want my options to fit my arch. I don't care if pip found x86-compatible one (probably shouldn't happen anymore)
Then, I'd want my options to fit my python-version: For py3.7, idk if pip found a requires-python:>=3.8
Finally, that's "the tricky part" for me: manylinux_2_17_ etc. Maybe pip should tell me "what kind of manylinux_2_17_ I support" (and finally, which versions it threw out)

My scenario here goes that I have 0 options, so pip can say that. Or, it can say those are your valid options.

It could be done so that some of those stages could be hidden (--dont-care-os), or the user could choose to re-arrange them (--resolution-order=python-version,os,version). As the stages should be "somewhat independent", chaining them differently (if only for diagnostic reasons) should not affect the end-end result - as all of those should be applied anyway.

Sep 20 '23 13:09 stdedos

@stdedos I'm pretty sure resolvelib doesn't provide an interface for changing the order of checks in the depresolver.

Sep 20 '23 14:09 webknjaz