Documenting specialisation rules

[Gobot1234]

I wasn't aware there were discrepancies between type checkers about this. Coming from https://github.com/microsoft/pyright/issues/5830, PEP 718 will require documentation about specialisation for functions and PEP 696 requires documentation about whether methods should bind default type parameters for the class.

PEP 718 problems

from typing_extensions import assert_type

class Foo[T, U]:
    def bar(self): ...

class Baz[U](Foo[int, U]):
    ...

# sorry for using PEP 677 syntax but it's easier to follow
# also Unknown is implicit Any (borrowed from pyright)
assert_type(Foo.bar, (self: Foo[Unknown, Unknown]) -> None)
assert_type(Baz.bar, (self: Foo[int, Unknown]) -> None)
assert_type(Foo[str, str].bar, (self: Foo[str, str]) -> None)

Foo[str, int].bar(Foo[str, int]())  # fine
Foo[int, int].bar(Baz[int]())  # fine
Baz.bar(Foo[str, int]())  # should error as Self is bound to Baz not Foo

MyPy

MyPy currently shows methods not binding type parameters at all which is problematic as type parameters should be bound in the scope they are defined. This behaviour is also I think incorrect in allowing the final call to pass as it is ignoring the specialisation of the class.

Pyright

Pyright currently shows, it isn't binding U to Unknown.

PEP 696 problems

from typing_extensions import assert_type

class Spam[T=int]:
    def meth[U](self, another: U, other: T) -> U:
        ...

assert_type(Spam.meth, (self: Spam[int], another: U, other: int) -> U)

MyPy

MyPy currently shows and appears to be binding T's default as I thought it should (good mind reading skills Marc), however, it is still suffering from the problems proposed above and would ignore any prior specialisation.

Pyright

Pyright currently shows meth as being partially unknown which is what I opened the original issue about.

A __new__ issue (🥁) W.R.T. PEP 718

Say I have a generic __new__/__init__ method that uses parameters that aren't bound by the class.

class New[T]:
    def __new__[U](cls, arg: T, l: list[U], elem: U):
        self = super().__new__(cls)
        l.append(elem)

This may seem a bit contrived but it can come up where there's a mapping between types e.g. for registering types for serialisation.

How should this be specialisable? A couple of options:

• New[T, U]() is fine: What about if it's added in init?
  • Class[ClassParams, ..., NewParams, ..., InitParams, ...]
  • Should order be lexicographical? I don't think there's a world in which this is practical
• Manually call through the __new__ method yourself and __init__ can't add any new type parameters.

My preference would be manually constructing the class through __new__ because the other case seems full of edge cases and is a special case with no real gain.

Summary of my thoughts

class Summary0[T, U]:
    def bar(self): ...

class Summary0Sub[U](Summary0[int, U]):
    ...

# Summary0 should bind type parameters in the scope they were defined
Summary0.bar  # should warn about implicit Any
# A specialised Summary0 should modify the type of self in methods
Summary0[int, bool].bar  # `self` should be Summary0[int, bool]
Summary0Sub[bool].bar  # `self` should be Summary0Sub[bool]

class Summary1[T=int]:
    def meth(self) -> T: ...

# defaults should bind on method access if not specialised
assert_type(Summary1().meth(), int)
assert_type(Summary1[str]().meth(), str)

class Summary2[T]:
    def id[U](self, x: U) -> tuple[T, U]: ...

assert_type(Summary2[bool]().id[str]("hi"), tuple[bool, str])
Summary2.id[int, str]  # error expected 1 type param not 2


class Summary3[T]:
    def __new__[U](cls, arg: T, l: list[U], elem: U): ...

Summary3[int].__new__[complex](1, [], 1j)
Summary3[int, complex](1, [], 1j)  # errors because special cases aren't special enough to break the rules

Does anyone have any objections to this? Where is this best documented neither PEP really feels like the right place for all of this.

Would be nice to hear from the MyPy and Pyright teams on this.

[mikeshardmind]

While I'm not currently maintaining a type checker, I agree with everything you have here except this:

Summary3[int, complex](1, [], 1j)  # errors because special cases aren't special enough to break the rules

I think this should be allowed and specified. If it's specified, it's no longer breaking the rules, it's part of the rules.

Edit: see below, but TLDR: this is unnecessary. if any author wants this to be their use, they can easily just make the function scoped type variable class scoped instead with minimal effort

[erictraut]

I think this should be allowed and specified

I strongly disagree with this. Summary3[int, complex] is illegal, and that's independent of what expression it's embedded within. The symbol Summary3 refers to a class that accepts a single type argument, and you are providing two type arguments here. Embedding it within some other expression doesn't make this legal.

Constructor calls in Python are not simple calls to __new__. They're much more complex. A constructor call invokes the metaclass' __call__ method and (typically) invokes the class' __new__ method and __init__ method (but there are cases where it doesn't). Any of these three methods can be generic and have function-scoped type parameters, and all of them can affect the final type of the constructed object. It therefore doesn't make sense to support explicit specialization of function-scoped type parameters in the __new__ method as part of a constructor call.

I'll note that function-scoped type parameters in __new__ and __init__ are relatively rare, so I don't see a strong need to provide a general solution to this problem. In the extremely rare case where it's needed, then manually calling through to __new__ seems like a fine workaround.

@Gobot1234, your examples above focus on instance methods. We also need to consider class methods and static methods. Do you think these should work consistently with instance methods? I could make arguments in either direction.

[mikeshardmind]

I think this should be allowed and specified

I strongly disagree with this. Summary3[int, complex] is illegal, and that's independent of what expression it's embedded within. The symbol Summary3 refers to a class that accepts a single type argument, and you are providing two type arguments here. Embedding it within some other expression doesn't make this legal.

Constructor calls in Python are not simple calls to __new__. They're much more complex. A constructor call invokes the metaclass' __call__ method and (typically) invokes the class' __new__ method and __init__ method (but there are cases where it doesn't). Any of these three methods can be generic and have function-scoped type parameters, and all of them can affect the final type of the constructed object. It therefore doesn't make sense to support explicit specialization of function-scoped type parameters in the __new__ method as part of a constructor call.

I'll note that function-scoped type parameters in __new__ and __init__ are relatively rare, so I don't see a strong need to provide a general solution to this problem. In the extremely rare case where it's needed, then manually calling through to __new__ seems like a fine workaround.

I mean, we're discussing writing a specification, saying it's illegal... well, it's only illegal if that's the direction of the specification. That said, you have persuasive reasons not to go this route here, In my opinion, the rarity of function scoped __new__ and __init__ is the significantly stronger argument, but there's another one I can add in this direction here too: Any currently function scoped argument could be made class scoped if that behavior is desired. In general, I don't find complexity-based arguments compelling because type-checking is only as useful as it is accurate, and almost every time a complexity argument is made, it is at the cost of accurately modeling the behavior, but in this case, there's an easy way to express the intended use for authors... just class scope if you need/want that use.

[Gobot1234]

@Gobot1234, your examples above focus on instance methods. We also need to consider class methods and static methods. Do you think these should work consistently with instance methods? I could make arguments in either direction.

Yes I think they should be consistent, I can't really think of any reasons to the contrary, especially because then you need to have more definitions for the whole descriptor protocol but happy to hear your thoughts on the matter.

[Gobot1234]

@JukkaL @ilevkivskyi @msullivan do any of you or anyone else on the mypy team have any opinions about changing the behaviour here for binding type params to and erasing them to AnyType(TypeOfAny.from_omitted_generics)