typing icon indicating copy to clipboard operation
typing copied to clipboard

[@overload]: What is the effect of the *implementation* type-signature?

Open alanhdu opened this issue 1 year ago • 3 comments

One thing that I've found underspecified when using @overload is what the type signature of the actual function should be.

For example, let's say I have a function like:

from typing import overload, Literal

@overload
def f(x: int, z: Literal[True]) -> str: ...
@overload
def f(x: int, z: Literal[False] = ...) -> int: ...
def f(x: int, z: bool = False) -> str | int:
    if z:
        return "hello"
    return 1

What is the effect of the last line? In particular:

  • The documentation and examples often just use no annotations for the implementation signature (def f(x, z):). Is that the right thing to do (I sometimes get Overloaded implementation is not consistent with signature of overload 1 errors if I leave out the type annotations)?
  • If I do specify types on the last, do I also have to add it to @overload list -- in other words:
@overload
def f(x: int, z: Literal[True]) -> str: ...
@overload
def f(x: int, z: Literal[False] = ...) -> int: ...
@overload
def f(x: int, z: bool = ...) -> str | int   # is this overload necessary?
def f(x: int, z: bool = False) -> str | int:
    if z:
        return "hello"
    return 1

Both pyright and mypy seem to interpret things differently with and without it (e.g. see this play link).

I understand that @overload is a complicated feature, but I'm hoping this is a small corner we can start with.

alanhdu avatar Aug 14 '24 23:08 alanhdu

The type signature of the implementation is never visible to callers. It is used for two things:

  • To type check the body the implementation
  • As a consistency check against the overloads (this is the "Overloaded implementation is not consistent with signature" error). Omitting type annotations should never trigger this, but things like omitting defaults (as in your example) or changing parameter names or kinds could

Regarding adding a third overload, I'm going to switch your example to using str for clarity:

@overload
def f(x: int, z: Literal["apple"]) -> str: ...
@overload
def f(x: int, z: Literal["banana"] = ...) -> int: ...
@overload
def f(x: int, z: str = ...) -> str | int  # what is the effect of this overload
def f(x: int, z: str = ...): ...

If you do not include the overload, type checkers will starting complaining on:

def main(z: str):
    f(0, z)

because they cannot guarantee the string is "apple" or "banana".

If you include the overload, type checkers will match that third overload on unknown strings.

bool makes things more interesting because there are no values other than True or False, so it would be valid for a hypothetical type checker to explode the bool automatically, making the third overload redundant. This is an area of overload resolution that would need to be specified to ensure all type checkers have that extra logic.

(Finally and more tangentially, I've found myself at times wanting an unsound literal feature (particularly so in typeshed). Where if the type checker knows the value of the literal, it uses the specific overloads, otherwise it aggregates information across the overloads)

hauntsaninja avatar Aug 15 '24 00:08 hauntsaninja

Do you mind if I file a PR to the docs (and maybe add a conformance test or two)? I think what you said makes sense, but just to check my understanding:

  • The implementation type-signature is never seen by callers (so if it has extremely broad types like setting everythin to Any, that has no effect on other callers)
  • That means you need to add an @overload for the "broadest type signature" as well -- just adding it for the implementation signature is not enough (this seems unfortunately redundant to me, but not the end of the world either).
  • For now, there's no special-casing of bool as only having 2 variants.

I think it's really the first point that I didn't understand. I assumed that the implementation type-signature was "automatically" counted as one of the overloads.

alanhdu avatar Aug 15 '24 16:08 alanhdu

@alanhdu, overloads are generally underspecified in the typing spec today. Earlier this week, I posted a proposed update to the "Overloads" chapter of the spec that attempts to fill in many of these gaps. It covers all of the points that you mention above (and many additional details). I welcome feedback on the proposed change. For example, if you think there are ways it could be clearer, please post comments to the thread or the PR.

If and when that update is ratified and incorporated into the typing spec, we'll make sure there are corresponding conformance tests. The current conformance tests for overloads are pretty anemic, reflecting the fact that the spec doesn't provide much guidance here.

The typing documentation site includes several different categories of documentation. My proposed update is to the typing specification, which is primarily targeted at type checker authors. In your points above, the wording might be more appropriate for the "Guides" style of documentation, which is primarily targeted at consumers of type checkers. I think that additional clarity about overloads within the "Guides" documentation would be a great contribution if you're interested in submitting a PR.

erictraut avatar Aug 15 '24 17:08 erictraut