"Could not resolve the type hint of" warning, for TypeVar

[giladbarnea]

Hi, thanks for the awesome library! Super useful.

I've been getting this warning:

Reproduce:

from typing import TypeVar

from plum import dispatch


class Foo:
    ...


FooClass = TypeVar("FooClass")


@dispatch
def bar(created_chat: int) -> int:
    return ...


@dispatch
def bar(
    foo_cls: FooClass = Foo,
) -> FooClass:
    return ...


bar()

[wesselb]

Hey @giladbarnea! Thanks for opening an issue. :)

Unfortunately type variables are currently not supported. :( It would be awesome to support them, but this is not a trivial undertaking.

If you ignore the warnings, your code will run, but it will not behave as expected:

from typing import TypeVar

from plum import dispatch


T = TypeVar("T")

@dispatch
def f(x: T) -> T:
    return 1

>>> f(2)   # OK so far...
1

>>> f("2")  # Nope! T_T This should error.
1

[giladbarnea]

Thanks for replying!

This is a bit of a bummer, if possible can you tell me a few reasons why supporting TypeVars is difficult? Hopefully I'll find some time and maybe try to fix it.

[wesselb]

@giladbarnea in principle supporting TypeVars should be possible, and I would really like to support them in the future. To do this, I think some kind of solver would have to be implemented that matches the type variables to concrete types. I've not thought super carefully about it, but this seems like a complex undertaking, since type variables can occur in all sorts of complicated nested types, e.g. Callable[dict[str, Tin], Treturn] -> Treturn.

[leycec]

Note to the unwise and the unwary: @beartype and thus Plum should superficially support a TypeVar with either:

• Explicit bounds (e.g., TypeVar('FooClass', bound=Foo)).
• Explicit constraints (e.g., TypeVar('FooClass', Foo)).

In either case, @beartype and thus Plum should reduce that TypeVar to that bounds and those constraints. This may require @beartype 0.16.0, which I'm on the precipice of releasing this Friday. Dare I do it? I dare.

As @wesselb suggests, full-blown type variable support at runtime is non-trivial in the extreme and not something anyone wants to voluntarily tackle. You'll either need full-time grant funding for that and/or Asperger's. I have the latter but not the former. Therefore, I'll eventually tackle this – but only after having exhausted every other outstanding feature request and issue in @beartype. It is the highest of the high-hanging fruit. It cannot be plucked without back pain, kidney pain, a blown ACL, and a ruptured Achilles heel. Don't go down that road, @giladbarnea.

[ilan-gold]

@leycec Are warnings expected with bound variables? They seem to work, but just want to make sure the warnings are expected and not from misuse.

from plum import dispatch
from typing import TypeVar

T = TypeVar("T", bound=str)

@dispatch
def f(x: T) -> T:
    return x

f('foo') # works with warnings
f(2) # errors, no warning

[wesselb]

@ilan-gold The warnings here can be ignored. Your code will run, but may not behave as expected.

Consider the following (very contrived) example:

from plum import dispatch
from typing import TypeVar


class Str2(str):
    pass

T = TypeVar("T", bound=str)

@dispatch
def f(x: T) -> T:
    return Str2(x)


f("hey")  # Works, but shouldn't, because the input is `str` and the output a `Str2`!

[sylvorg]

f("hey") # Works, but shouldn't, because the input is str and the output a Str2!

Doesn't that work just because Str2 is a subclass of str?

[wesselb]

Right, I see what you're saying. I believe that the way T is supposed to work is that it binds to exact types.

Consider instead the following example:

from plum import dispatch
from typing import TypeVar


class Str2(str):
    pass

T = TypeVar("T", str, int)

@dispatch
def f(x: T) -> T:
    return 1


f("1")  # Works, but shouldn't, because the input is `str` and the output a `int`!

[sylvorg]

Wait, this time, aren't the constraints functioning as a union of types now, though?

[wesselb]

Wait, this time, aren't the constraints functioning as a union of types now, though?

That's what's currently happening, yes, but it's not the correct behaviour.

The above code should be equal to

@dispatch
def f(x: str) -> str:
    ...

@dispatch
def f(x: int) -> int:
    ...

It would be great to properly support type parameters, but unfortunately that's not an easy feat.

[sylvorg]

Why not get the type of the argument passed to the function, find the index of the type in the constraint, then check the index of the return type in the return constraint? With special consideration of exact types. Like:

def check(func, arg, argvar, _return, returnvar):
    argtype = type(arg)
    returntype = type(_return)
    for i, t in enumerate(returnvar.__constraints__):
        if argtype is t and argvar.__contraints__[i] is t:
            return True
    for i, t in enumerate(returnvar.__constraints__):
        if issubclass(argtype, t) and issubclass (argvar.__contraints__[i], t):
            return True
    return False

Of course, this is just a short mock-up!

[wesselb]

@sylvorg, you're totally right that this would be a nice attempt at supporting type parameters. Currently, we don't do any of this.

Should you want to have a go at trying to properly support type parameters, then that would be super exciting. I think the basic cases can be covered in a fairly straightforward way. Getting all the edge cases will likely be tedious and very difficult.

[sylvorg]

I'll try my best, but it'll take me a while to understand the code base; which files would you recommend I look at, other than types.py?

[wesselb]

To be honest, this might be a pretty major undertaking that touches a large part of the codebase. It's currently not clear to me what the best way of going about it would be.

I'm thinking that we could add a "type parameter resolution stage" in resolver.py, where Methods with type parameters are converted into Methods without type parameters. Not entirely sure.

[sylvorg]

Is there any function in the codebase you can think of that gets the argument type, the argument annotation, the return type, and the return annotation? Or a series of functions? I could start working from there. Or would they all be in resolver.py?

[wesselb]

Function in function.py collects all Methods (method.py). Here a method is defined as an implementation of a function for a type signature.

The actual types of the arguments are only considered at the very last stage of dispatch, in resolver.py. Resolver chooses which Method is appropriate for the given arguments. I think that's the point where type parameters could be handled.

Thinking about it, I think all that might be required is a matching algorithm that attempts to determine the values of the type parameters (the first hit suffices) and which then replaces that type parameter by the matched value.

[sylvorg]

Thinking about it, I think all that might be required is a matching algorithm that attempts to determine the values of the type parameters (the first hit suffices) and which then replaces that type parameter by the matched value.

Sorry, could you expand on this a little bit? What do you mean by "replaces that type parameter by the matched value"?

[wesselb]

@sylvorg, basically, if the signature is (list[T], int, T) -> T and the arguments are ([1], 5, "test"), then, by matching list[T] to [1], it is clear that T = int. Therefore, we can substitute int for T, giving the "concrete" signature (list[int], int, int) -> int, and we can use the existing machinery on that concrete signature.

[sylvorg]

Oof; this is getting a little too confusing for me... 😅 Would we not also have to match the type of the return value as well as the arguments provided to the function?

[wesselb]

@sylvorg, yes, that's completely right, and it's one of the main challenges why this is so difficult. :( In general, signatures can depend on T in arbitrarily complex ways, and you need a generic mechanism that can infer the value of T for arbitrary arguments.

We could decide to take it step by step and only support type parameters in a limited manner. If we code things up in a robust and sound way and give appropriate warnings to the user, I would be fully on board with that.

[leycec]

Indeed. @wesselb has the right of it. @wesselb always does. @beartype itself will (...probably) begin tackling type parameters in earnest sometime in 2025. The plan here is exactly as @wesselb briskly delineated: "take it step by step and only support type parameters in a limited manner." In order, @beartype will begin supporting callable signatures annotated by:

1. First, mandatory positional-only root type parameters (i.e., type parameters annotating only mandatory positional-only parameters not subscripting other type hints): e.g.,

   def mutha_funcer[T](/, muh_arg: T, nuther_arg: T, hooboy_arg: list[set[int]]) -> T: ...
   

   Here, the type parameter T only appears in its root unsubscripted form. Since all parameters are both mandatory and positional-only, T is guaranteed to bind to the type of the first parameter; all subsequent instances of T are then constrained to be of the same type. Other unrelated type hints like hooboy_arg also appear and are handled in an unrelated customary way.

2. Next, mandatory flexible root type parameters (i.e., type parameters annotating only mandatory parameters that may be passed either positionally or by keyword that are not subscripting other type hints): e.g.,

   def badboi_func[T](muh_arg: T, nuther_arg: T, hooboy_arg: list[set[int]]) -> T: ...
   

   Again, the type parameter T only appears in its root unsubscripted form. Although all parameters are mandatory, parameters may now be passed either positionally or by keyword. This means that the first passed parameter is unknown at decoration time. Above, the first passed parameter is guaranteed to be muh_arg. Here, however, the first passed parameter could be either muh_arg, nuther_arg, or hooboy_arg if all three are passed by keyword. Flexibility is thus paramount. Whichever parameter is passed first, T now dynamically binds to the type of that parameter; all subsequent instances of T are then constrained to be of the same type.

And so on and so forth. It's super-fun just to cogitate, ideate, and ruminate about this. We're deep into the Philosophy of Type Hints at this point. But it's also super-critical to have a coherent plan of where to begin. Start with the absolute simplest use case and cautiously build out support for increasingly less simplistic use cases from there – iteratively ratcheting up the stakes like a sweaty late-night game of spin-the-bottle that can only end in disaster for all parties.