TypeVar substitution in get_type_hints

[wyfo]

When it comes to generic classes, type annotations retrieved with get_type_hints are not really easy to use, as in the following example:

from typing import Generic, TypeVar, get_type_hints

T = TypeVar("T")
class A(Generic[T]):
    a: T

U = TypeVar("U")
class B(A[U]):
    b: U

assert get_type_hints(B) == {"a": T, "b": U}

It could be useful to add a TypeVar substitution to get_type_hints, for example with an additional substitute_type_vars parameter (with a False default) By the way, get_type_hints could also allow to pass generic alias in order to substitute directly TypeVars with their related argument.

It would give for the example above:

assert get_type_hints(B, substitute_type_vars=True) == {"a": U, "b": U}  # typevars are consistent
assert get_type_hints(B[T], substitute_type_vars=True) == {"a": T, "b": T}
assert get_type_hints(B[int], substitute_type_vars=True) == {"a": int, "b": int}  # ready to use

[wyfo]

Here is a POC of implementation, using the feature of generic MRO described in #777:

import sys
from typing import *
from typing import _collect_type_vars, _eval_type, _strip_annotations

##### From #777
def _generic_mro(result, tp):
    origin = get_origin(tp)
    if origin is None:
        origin = tp
    result[origin] = tp
    if hasattr(origin, "__orig_bases__"):
        parameters = _collect_type_vars(origin.__orig_bases__)
        if origin is tp and parameters:
            result[origin] = origin[parameters]
        substitution = dict(zip(parameters, get_args(tp)))
        for base in origin.__orig_bases__:
            if get_origin(base) in result:
                continue
            base_parameters = getattr(base, "__parameters__", ())
            if base_parameters:
                base = base[tuple(substitution.get(p, p) for p in base_parameters)]
            _generic_mro(result, base)

def generic_mro(tp):
    origin = get_origin(tp)
    if origin is None and not hasattr(tp, "__orig_bases__"):
        if not isinstance(tp, type):
            raise TypeError(f"{tp!r} is not a type or a generic alias")
        return tp.__mro__
    # sentinel value to avoid to subscript Generic and Protocol
    result = {Generic: Generic, Protocol: Protocol}
    _generic_mro(result, tp)
    cls = origin if origin is not None else tp
    return tuple(result.get(sub_cls, sub_cls) for sub_cls in cls.__mro__)
#####

def _class_annotations(cls, globalns, localns):
    hints = {}
    if globalns is None:
        base_globals = sys.modules[cls.__module__].__dict__
    else:
        base_globals = globalns
    for name, value in cls.__dict__.get("__annotations__", {}).items():
        if value is None:
            value = type(None)
        if isinstance(value, str):
            value = ForwardRef(value, is_argument=False)
        hints[name] = _eval_type(value, base_globals, localns)
    return hints


# For brevety of the example, the implementation just add the substitute_type_vars
# implementation and default to get_type_hints. Of course, it would have to be directly
# integrated into get_type_hints
def get_type_hints2(
    obj, globalns=None, localns=None, include_extras=False, substitute_type_vars=False
):
    if substitute_type_vars and (isinstance(obj, type) or isinstance(get_origin(obj), type)):
        hints = {}
        for base in reversed(generic_mro(obj)):
            origin = get_origin(base)
            if hasattr(origin, "__orig_bases__"):
                parameters = _collect_type_vars(origin.__orig_bases__)
                substitution = dict(zip(parameters, get_args(base)))
                annotations = _class_annotations(get_origin(base), globalns, localns)
                for name, tp in annotations.items():
                    if isinstance(tp, TypeVar):
                        hints[name] = substitution.get(tp, tp)
                    elif tp_params := getattr(tp, "__parameters__", ()):
                        hints[name] = tp[
                            tuple(substitution.get(p, p) for p in tp_params)
                        ]
                    else:
                        hints[name] = tp
            else:
                hints.update(_class_annotations(base, globalns, localns))
        return (
            hints
            if include_extras
            else {k: _strip_annotations(t) for k, t in hints.items()}
        )
    else:
        return get_type_hints(obj, globalns, localns, include_extras)

This implementation has been tested with Generic but also with PEP 585 builtin generic containters.

[gvanrossum]

IMO subtleties related to type variables are best left to static type checkers. What's the use case that led you down this path?

[wyfo]

Actually, this code is (almost) directly extracted from one of my library: apischema

Type annotations are not exclusively used by type checkers, that's what makes their strength. In my library, i use type annotations to generate (de)serializers/JSON schema/GraphQL schema/etc. just from type annotations.

And this is actually a (very) big use case : pydantic and thus FastAPI are literally built on this.

In all of these libraries, generics are not so easy to handle (pydantic has even gave up to support them in 3.6) and it requires a lot of code to be done properly; you have a direct example with my code above.

[gvanrossum]

But you can do these calculations yourself using other public APIs defined in typing.py, right? I am not in favor of complexifcation of get_type_hints() for such a corner case (even though I understand it's important to you).

[wyfo]

But you can do these calculations yourself using other public APIs defined in typing.py, right?

It requires a little bit of private (_eval_type), but yes it can be done.

Acutally, I doesn't need this feature in typing for myself, but I thought it could be good to make easier the creation of new tools using type annotations (and FastAPI skyrocketing shows that there is a demand). Maybe a library on Pypi like typing_inspect would be the best way in the end.

[devdoomari3]

@wyfo I've been working on extracting out python's type-annotations in runtime: python-type-extractor

To see how it works, look into files in test_fixtures folder folder, and the corresponding test files in type_extractor/types folder

For example, this sample code rel. generics is extracted to this test output

you can use the library however you want (it's "do whatever you want just add a small credit on readme" license... ), and file issues if you have any problems.
(btw I'm looking for the next maintainer for that library 😉 -- I'm not much of a python-person) Also, based on this lib, I was working on something similar to your https://github.com/wyfo/apischema -- but using codegen. (current code is somewhat messy -- I'm waiting for input-union

@gvanrossum I think having a better 'annotation-processing' should be a priority for the python ecosystem. A lot of type-safe languages try to provide some form of annotation-processing tools, which is then used to 'extend' the language in various ways (eg. code-generations):

• for java, it's 'annotation-processor' -- see https://github.com/gunnarmorling/awesome-annotation-processing
• for Kotlin, there's KAPT
• for Rust, there's Procedural macro (though this is based on ASTs, there are a lot of helper libs to make it usable like annotation-processing tool)
  • used to augment language itself
    • eg. before the official async-await syntax, there were unofficial libs as temp alternative
    • eg. now, since rust doesn't have async-fn trait officially, there's a lib for providing that

Also, some standard build-time API-hook that can be invoked by something like pip run build and auto-invoked by IDEs will be great 🤗

[gvanrossum]

I have to be honest and direct, I am already spread too thinly. You need to find someone else to champion this.

[wyfo]

@devdoomari3 Your self-promotion is too much insistent, it's quite embarrassing. By the way, your suggestion of static evaluation, even interesting, seems to me quite off-topic in this thread.

Briefly, one strength of Python is that you don't need core language features/syntax to do a lot of manipulations of pseudo-static elements like annotations (while you made a comparison with languages where annotations are truly static things).

Your concern is about IDE handling of this manipulations (and I understand it), but you could write a plugin for your favorite one which will execute your code and adapt his typeshed from the result. No need to change the typing module.

[devdoomari3]

@wyfo it's not static-evaluation -- it's runtime type extraction, which is what you're trying to do.
-- I'm hoping someone else will take interest and pick it up ( I'm moving away from python... ) -- or someone who knows python better gets the idea, and start a better version of it (there's a large room for improvement -- perf-wise, output-wise, quality-wise)

As for IDE handling, I'm hoping for some standardization for runtime-annotation-processing or codegen: \

• for Dart lang, a file with *.g.dart is assumed to be generated
• for java/kotlin, anything in intermediate/ folder is assumed to be generated

by having some standard, we can get:

• IDEs showing a do-not-edit warning sign for those files
• standard gitignore including those files
• standard way of triggering code generation
• (optional) source map of which original code line is relevant

Another thing: because there are multiple implementations of typecheckers, there's the problem of 'plugins only written for mypy not working for pytype / pylance / etc' --- extremely useful things like sqlalchemy-mypy plugin, pydantic-mypy plugin only works for mypy... -- this is something python can solve with standardized code-generation tools

Anyway, I don't know a lot about python / how python decisions are made / writing IDE plugin, and how much I should prepare to start a PEP (is it ok to have only rough draft? will others with better python knowledge fill in the gaps? etc)