Typing Usability Overhaul

[ambv]

While I am a big proponent of static typing, I always found typing, the module, clunky. And typing_extensions? That thing I just hate, it's a usability disaster. In an ideal world, people wouldn't need those extra modules, at least not for the common things they want to do. How can we get there?

Accept PEP 563

It lets people get away with not having a runtime typing for annotations. Read below for edge cases.

Split typing

The goal here is to minimize the need for people to import from typing at all.

Move NamedTuple to collections

It always belonged there.

We need TYPE_CHECKING in the interpreter

Let's call it __typing__. That way we can really remove the need for a runtime typing module, unless somebody creates generic classes. Thanks to PEP 563 that would be the new idiom:

if __typing__:
    from typing import *
    T = TypeVar('T')
    UserId = NewType('UserId', int)
    ...  # etc.

def f(users: List[UserId]) -> Optional[UserId]:
    ...

For context, I would rename TYPE_CHECKING to __typing__ just like __debug__ because I would also like to be able to flip the switch to be True with a command-line option/env variable to Python. That is required to make people wanting runtime type checking happy.

cast() needs to change

I suggest this syntactic change so that we don't need to add it to Python's builtins:

# reveal_type(previous_result) is `builtins.int`
result: cast(UserId) = previous_result

That way we don't actually cast anything at runtime, there's no function call. While it doesn't compose to random expressions and you need a new artificial name for this, I find this an improvement.

get_type_hints() needs to move to inspect

Untying it from the logic in typing will be a bit of a challenge but this is required if PEP 563 is to be accepted at all. We need a blessed way in Python proper to evaluate annotations at runtime.

ABCs belong in collections.abc

It always felt ugly to me that we have separate typing equivalents for collections. Builtin types, that I get, but having a separate Counter, and better yet, MutableMapping, that feels wrong.

I would add generics support to collections directly and make typing just mirror those real classes in 3.7+ for backwards compat. The damage is already done, people will try to import those types from typing.

The rest goes to a PyPI module called typing

This includes the current and future contents of typing_extensions and ideally also mypy_extensions.

PEP 563 deep dive

For now, if __typing__: from typing import * will do. But that's clunky and wordy. In the future, PEP enables more friendly syntax that doesn't require typing at all, like list[str], collections.Mapping[int, int], any, str or None, etc. This needs a separate PEP and a ton of bike-shedding, but if the point is to make type annotations feel more first-class, we need to acknowledge that we can and should improve the common case to be more obvious to users.

What if you really need typing at runtime?

That's okay. If you subclass generics, or want to use any other thing that cannot be put in annotations or if __typing__:, then add typing to your requirements.txt and import it as a regular module.

[gvanrossum]

Whoa, that's a couple more PEPs almost.

NamedTuple

I looked at the code for NamedTuple and it currently depends on a whole lot of other stuff in typing. Also IIRC there are still some improvements possible (though perhaps those should be implemented in mypy, not in the runtime support).

__typing__

Sure, that's easy, one new builtin that's always False.

cast()

Interesting idea, but probably needs some more discussion, and I don't want to start new projects this close to the deadline.

get_type_hints()

This concerns me, because with PEP 563 around the corner I expect there will be a bunch more work on this to fix corner cases. (It's remarkably complicated.) So this is actually an area where I'd have hoped the non-stdlib typing.py would allow us to do better.

ABCs

Again, these currently depend on a fair amount of other stuff in typing.py (in particular Generic and GenericMeta).

PEP 563 deep dive

This section is incomprehensible (probably written in a hurry). You're proposing yet another PEP? Is this relevant at all for Python 3.7?

---

In general I'm not excited by the idea of trying to address all these in 3.7. If we can't move typing.py out of the stdlib, so be it. But let's not rush into a bunch of redesign just to enable it. There's always 3.8.

[ambv]

You're right that there's too little time for an overhaul now. Let's table this, I gave smaller scale suggestions back in #495.

[johnthagen]

If we can't move typing.py out of the stdlib, so be it

We have a bunch of 3.5+ production code that imports typing that for various reasons cannot depend on PyPI packages. If typing is suddenly removed from the stdlib in 3.7 or 3.8, it would break a bunch of our code. We must have misunderstood what "provisional" meant, and thought it meant things could still be added or tweaked in minor releases (3.5.1, 3.5.2, etc), and didn't realize it could mean the entire module could be removed from the stdlib.

We don't usually use very advanced typing features, but the basic List, Dict, etc generic types are invaluable for day to day increased productivity in PyCharm. For us, the basics that exist in 3.5.0+ get the job done and so we aren't as effected by needing the "latest and greatest" typing features.

[gvanrossum]

@johnthagen You're safe here, we decided to keep it in the stdlib.

But I recommend that you read PEP 411 carefully next time -- it explicitly says "up to and including removal of the package". The Python docs say this too at https://docs.python.org/3/glossary.html#term-provisional-api, which is linked from the provisional warning at the top of the docs for the typing module (https://docs.python.org/3/library/typing.html?highlight=typing%20tuple).

There are other words on all those pages suggesting that this is only a "solution of last resort". And your use case would have given me thought if we hadn't already decided to keep typing in the stdlib.. (And no, we're not going through the same scare scenario when 3.8 comes around.)

[johnthagen]

@gvanrossum Thanks for the quick response and links. We'll be sure to fully understand the nature of provisional modules and APIs in future use.

At the end of the day, we understand there are trade-offs between development speed and stability and why provisional modules were added. If something like removing typing did really need to be done for the good of all future Python, we'd understand. We are glad that it sounds like there is a path forward without losing the backwards compatibility in this case.

Type hinting in Python is a major productivity win for us, so thanks for all of the work that's gone into it.

[sametmax]

To add to the wishlist:

• accept callable instead of typing.Callable
• type[] instead of typing.Type[]

[JelleZijlstra]

I'm going to close this. Many of the proposals in the original post have been implemented. Others, like building TYPE_CHECKING into the interpreter, haven't been, but they don't appear to be widespread pain points.

If any specific feature is still desired, please open a more focused issue.