mypy
mypy copied to clipboard
python
TypeForm[T]: Spelling for regular types (int, str) & special forms (Union[int, str], Literal['foo'], etc)
(An earlier version of this post used TypeAnnotation rather than TypeForm as the initially proposed spelling for the concept described here)
Feature
A new special form TypeForm[T] which is conceptually similar to Type[T] but is inhabited by not only regular types like int and str, but also by anything "typelike" that can be used in the position of a type annotation at runtime, including special forms like Union[int, str], Literal['foo'], List[int], MyTypedDict, etc.
Pitch
Being able to represent something like TypeForm[T] enables writing type signatures for new kinds of functions that can operate on arbitrary type annotation objects at runtime. For example:
# Returns `value` if it conforms to the specified type annotation using typechecker subtyping rules.
def trycast(typelike: TypeForm[T], value: object) -> Optional[T]: ...
# Returns whether the specified value can be assigned to a variable with the specified type annotation using typechecker subtyping rules.
def isassignable(value: object, typelike: TypeForm[T]) -> bool: ...
Several people have indicated interest in a way to spell this concept:
- @ltworf: https://github.com/python/mypy/issues/9003
- @glyph: https://github.com/python/mypy/issues/9003#issuecomment-653353318
- @davidfstr (yours truly): https://github.com/python/mypy/issues/9003#issuecomment-734648129
- @Nico-Kialo: https://github.com/python/mypy/issues/8992
- @hmvp: https://github.com/python/mypy/issues/8992#issuecomment-647331625
For a more in-depth motivational example showing how I can use something like TypeForm[T] to greatly simplify parsing JSON objects received by Python web applications, see my recent thread on typing-sig:
If there is interest from the core mypy developers, I'm willing to do the related specification and implementation work in mypy.
Why not do something like:
from __future__ import annotations
from typing import *
T = TypeVar("T")
class TypeAnnotation(Generic[T]):
@classmethod
def trycast(cls, value: object) -> T:
...
reveal_type(TypeAnnotation[Optional[int]].trycast(object()))
Why can't we make Type[T] also work for other special forms?
@hauntsaninja's workaround is useful, but it would be better to have a feature in the core type system for this.
@hauntsaninja , using your workaround I am unable to access the passed parameter at runtime from inside the wrapper class.
The following program:
from __future__ import annotations
from typing import *
T = TypeVar('T')
class TypeAnnotation(Generic[T]):
@classmethod
def trycast(cls, value: object) -> Optional[T]:
Ts = get_args(cls)
print(f'get_args(cls): {Ts!r}')
return None
ta = TypeAnnotation[Union[int, str]]
print(f'get_args(ta): {get_args(ta)!r}')
result = ta.trycast('a_str')
prints:
get_args(ta): (typing.Union[int, str],)
get_args(cls): ()
@JelleZijlstra commented:
Why can't we make
Type[T]also work for other special forms?@hauntsaninja's workaround is useful, but it would be better to have a feature in the core type system for this.
I agree that having Type[T] be widened to mean "anything typelike, including typing special forms" would be an alternate solution. A very attractive one IMHO.
However it appears that there was a deliberate attempt in mypy 0.780 to narrow Type[T] to only refer to objects that satisfy isinstance(x, type) at runtime. I don't understand the context for that decision. If however that decision was reversed and Type[T] made more general then there would be no need for the additional TypeAnnotation[T] syntax I'm describing in this issue.
I believe the issue is that Type is used for things that can be used as the second argument of isinstance(). And those things must be actual class objects (or tuples of such) -- they cannot be things like Any, Optional[int] or List[str].
So if this feature is going to happen I think it should be a separate thing -- and for the static type system it probably shouldn't have any behavior, since such objects are only going to be useful for introspection at runtime. (And even then, how are you going to do the introspection? they all have types that are private objects in the typing module.)
Well I wrote this module with various checks and add every new major py version to the tests to see that it keeps working: https://github.com/ltworf/typedload/blob/master/typedload/typechecks.py
Luckily since py3.6 it has not happened that the typing objects change between minor upgrades of python.
I believe the issue is that Type is used for things that can be used as the second argument of isinstance(). And those things must be actual class objects (or tuples of such) -- they cannot be things like Any, Optional[int] or List[str].
Makes sense.
for the static type system it probably shouldn't have any behavior, since such objects are only going to be useful for introspection at runtime.
Agreed.
how are you going to do the introspection? they all have types that are private objects in the typing module.
The typing module itself provides a few methods that can be used for introspection. typing.get_args and typing.get_origin come to mind.
I could see a couple of possible spellings for the new concept:
-
TypeAnnotation(as proposed earlier by the issue title) -
TypeForm-- as in a "typing special form", which is consistent with the following existing documentation:- "special forms" -- https://docs.python.org/3/library/typing.html#special-forms
- "special typing form" -- https://docs.python.org/3/library/typing.html#typing.get_origin
-
TypeHint-- as in the original PEP 484 "Type Hints", although I still lean toward "type annotation" as more of the proper full name
Personally I'm now leaning toward TypeForm (over TypeAnnotation) because it is consistent with prior documentation and is more succinct to type. It does sound a bit abstract but I expect only relatively advanced developers will be using this concept anyway.
(Let the bikeshedding begin. :)
Okay I'll go with TypeForm then, for lack of other input.
Next steps I expect are for me to familiarize myself with the mypy codebase again since I'm a bit rusty. Hard to believe it's been as long as since 2016 I put in the first version of TypedDict. Rumor is it that the semantic analyzer has undergone some extensive changes since then.
Good. And yeah, a lot has changed. Once you have this working we should make a PEP out of it.
Once you have this working we should make a PEP out of it.
Yep, will do this time around. :)
Update: I redownloaded the high-level mypy codebase structure this afternoon to my brain. It appears there are now only 4 major passes of interest:
-
State.semantic_analysis_pass1() -
semanal_main.semantic_analysis_for_scc() # pass 2 -
TypeChecker.check_first_pass() -
TypeChecker.check_second_pass()
Next steps I expect are to trace everywhere that mypy is processing occurrences of Type[T] and T = TypeVar('T'), which I expect to be most-similar in implementation to the new TypeForm[T] support.
Update: I have found/examined all mypy code related to processing the statement T = TypeVar('T'). Briefly:
- A TypeVarExpr(TypeVarLikeExpr) is parsed from the characters
T = TypeVar('T')bymypy.fastparse.parse(). - A TypeVarDef(TypeVarLikeDef) is parsed from [a TypeVarExpr assigned to a name] and put into the current scope by
TypeVarLikeScope.bind_new(name: str, TypeVarLikeExpr) -> TypeVarLikeDef. - A TypeVarType is returned as the resolved type for an unbound type reference by
TypeAnalyser.visit_unbound_type_nonoptional(t: UnboundType, ...) -> Type
Next steps I expect are to trace everywhere that mypy is processing occurrences of Type[T] and other references to a T (which a TypeVar assignment statement defines).
Update: I did trace everywhere that mypy is processing occurrences of Type[T], and more specifically uses of TypeType. There are a ton!
In examining those uses it looks like the behavior of TypeForm when interacting with other type system features is not completely straightforward, and therefore not amenable to direct implementation. So I've decided to take a step back and start drafting the design for TypeForm in an actual PEP so that any design issues can be ironed out and commented on in advance.
Once it's ready, I'll post a link for the new TypeForm PEP draft to here and probably also to typing-sig.
Yeah, alas Type[] was not implemented very cleanly (it was one of the things I tried to do and I missed a lot of places). We do have to consider whether this is going to be worth it -- there are much more important things that require our attention like variadic generics and type guards.
We do have to consider whether this is going to be worth it -- there are much more important things that require our attention like variadic generics and type guards.
Aye. Variadic generics and type guards both have much wider applicability than TypeForm in my estimation. Python 3.10's upcoming alpha window from Feb 2021 thru April is coming up fast, and only so many PEPs can be focused on.
Nevertheless I'll get the initial TypeForm PEP draft in place, even if it needs to be paused ("deferred"?) for a bit.
Update: I've drafted an initial PEP for TypeForm.
However I was thinking of waiting to post the TypeForm PEP for review (on typing-sig) until the commenting on PEP 646 (Variadic Generics) slows down and it becomes soft-approved, since that PEP is consuming a lot of reviewer time right now and is arguably higher priority.
In the meantime I'm building out an example module (trycast) that plans to use TypeForm.
Update: I'm still waiting on Variadic Generics (PEP 646) on typing-sig to be soft-approved before posting the TypeForm PEP draft, to conserve reviewer time.
(In the meantime I'm continuing to work on trycast, a new library for recognizing JSON-like values that will benefit from TypeForm. Trycast is about 1-2 weeks away from a beta release.)
I expect PEP 646 to be a complex topic to soft-approve, given the complexity of implementation, so I recommend not blocking on that for too long (though waiting a little while longer is fine).
I've posted the draft of the TypeForm PEP to typing-sig for review and discussion at: https://mail.python.org/archives/list/[email protected]/thread/7TDCBWT4RAYDJUQJ3B5NKXTQDUO5SIW2/
Update: Did give a presentation about TypeForm at PyCon US 2021 with representatives from most type checkers attending. The folks in the room (including Jukka from mypy) were leaning toward broadening Type[T] to also match non-class types (and type annotation objects in general) in preference to introducing an entirely new form like TypeForm[T]. This would avoid the need to introduce a new spelling that must be remembered (TypeForm) and likely be easier to implement. It sounded like some type checkers other than mypy already have this behavior.
So the next steps I foresee is drafting a proposal to broaden Type[T] as defined by PEP 484 to accept any type annotation object and not just class objects.
On my own plate I'm still pushing the implementation of PEP 655, and I'm planning to switch back to TypeForm discussions/work after that is complete.
The folks in the room (including Jukka from mypy) were leaning toward broadening
Type[T]to also match non-class types (and type annotation objects in general) in preference to introducing an entirely new form likeTypeForm[T].
This change would "break" some of my code. typing.Type was designed as a wrapper of builtin type, but type has some characteristics (__name__, __mro__, etc.) that are not present in all possible types, e.g. typing.TypeVar, typing.NewType or generic aliases.
So if I have variable annotated with Type on which I access __mro__, this modification of PEP 484 would make this access unsafe, and I would have to add # type: ignore everywhere I use my variable as a type.
(I know I could use type instead of Type, especially since Python 3.9, but my code must be Python 3.6-compatible, and having a different meaning between type and Type is kind of awkward to me)
I don't really like this kind of modification with breaking impact. By the way,typing.Type is supposed to be deprecated since Python 3.9 because of PEP 585. This would mean to remove the deprecation.
Was this issue addressed during the PyCon?
(By the way, I've always wanted to say that I find AnyType more explicit than TypeForm)
@wyfo do not rely on dunder methods, as they can (and do) change between versions; in some cases even between minor releases.
See the links. The internal APIs of the typing stuff has had a lot of changes in the various releases. If the specific stuff you used happened to not be changed I'd attribute it more to luck than anything else.
https://github.com/ltworf/typedload/blob/1.19/typedload/typechecks.py#L105-L110
https://github.com/ltworf/typedload/blob/master/typedload/typechecks.py#L63-L67
https://github.com/ltworf/typedload/blob/master/typedload/typechecks.py#L79-L86
https://github.com/ltworf/typedload/blob/master/typedload/typechecks.py#L99-L108
I know that typing API is very instable (i've written myself adaptors for cross-version typing.get_type_hints, typing.get_origin, typing.get_args, etc.), however, my point is not about typing API, it's about builtin object type and its instances.
And I do think all dunder attributes (__mro__, __init__, etc.) described in the data model documentation are pretty stable. So when I have a Type object instance, a.k.a. a class, I can expect to have __mro__ available; it would not be the case with an instance of NewType. There is no luck here.
Yup. And this is why I still prefer keeping Type[] for things that are actual class objects and introducing TypeForm[] for other things that have concrete classes defined in typing.py.
On Thu, May 27, 2021 at 2:20 AM wyfo @.***> wrote:
I know that typing API is very instable (i've written myself adaptors for cross-version typing.get_type_hints, typing.get_origin, typing.get_args, etc.), however, my point is not about typing API, it's about builtin object type and its instances.
And I do think all dunder attributes (mro, init, etc.) described in the data model documentation https://docs.python.org/3/reference/datamodel.html are pretty stable. So when I have a Type object instance, a.k.a. a class, I can expect to have mro available; it would not be the case with an instance of NewType. There is no luck here.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/python/mypy/issues/9773#issuecomment-849479377, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAWCWMQ745BQ4LZ62QMDS43TPYFG7ANCNFSM4UIO3ZYA .
-- --Guido van Rossum (python.org/~guido)
By the way,typing.Type is supposed to be deprecated since Python 3.9 because of PEP 585. This would mean to remove the deprecation.
Was this issue addressed during the PyCon?
If the meaning of Type[T] was changed I would presume that type[T] would also change since the latter is an alternative spelling for the former.
On Thu, May 27, 2021 at 7:52 PM David Foster @.***> wrote:
By the way,typing.Type is supposed to be deprecated since Python 3.9 because of PEP 585. This would mean to remove the deprecation.
Was this issue addressed during the PyCon?
If the meaning of Type[T] was changed I would presume that type[T] would also change since the latter is an alternative spelling for the former.
Indeed.
-- --Guido van Rossum (python.org/~guido)
If the meaning of Type[T] was changed I would presume that type[T] would also change since the latter is an alternative spelling for the former.
But I don't think we can change the meaning of type[T], because type is a builtin object; changing it would be like saying that list[T] could be something else than a list.
But I don't think we can change the meaning of
type[T], becausetypeis a builtin object; changing it would be like saying thatlist[T]could be something else than alist.
Actually it works just fine. You can define a function with an argument whose type is said to be type:
def f(t: type): ...
and you can call it with e.g. a union type:
f(int | str)
Inside f we could introspect the argument and pick it apart, e.g.:
import types
def f(t: type):
match t:
case types.Union(__args__=a):
print(a)
And the above call f(int | str) would print this:
(<class 'int'>, <class 'str'>)
Adding a typevar to the annotation makes no difference.