python
Making a type alias of a Callable that describes a decorator discards all type information
Bug Report
Attempting to make a type alias for a decorator results in MyPy losing type information.
To Reproduce
from typing import *
type Fn[**P, R] = Callable[P, R]
type Decorator[**P, R] = Callable[[Fn[P, R]], Fn[P, R]]
def do_the_thing[**P, R]() -> Decorator[P, R]:
def decorator(fn: Fn[P, R]) -> Fn[P, R]:
return fn
return decorator
@do_the_thing()
def blahblah() -> None:
pass
Seems this is not an artifact of how variance is being calculated with the new type alias syntax as using the old style syntax has the same issue when declaring the type hints...
from typing import *
P = ParamSpec("P")
R_co = TypeVar("R_co", covariant=True)
Fn: TypeAlias = Callable[P, R_co]
Decorator: TypeAlias = Callable[[Fn[P, R_co]], Fn[P, R_co]]
def do_the_thing[**P, R_co]() -> Decorator[P, R_co]:
def decorator(fn: Fn[P, R_co]) -> Fn[P, R_co]:
return fn
return decorator
@do_the_thing()
def blahblah() -> None:
pass
...and the same issue if I just discard the type variables entirely in the function definition:
from typing import *
P = ParamSpec("P")
R_co = TypeVar("R_co", covariant=True)
Fn: TypeAlias = Callable[P, R_co]
Decorator: TypeAlias = Callable[[Fn[P, R_co]], Fn[P, R_co]]
def do_the_thing() -> Decorator[P, R_co]:
def decorator(fn: Fn[P, R_co]) -> Fn[P, R_co]:
return fn
return decorator
@do_the_thing()
def blahblah() -> None:
pass
Using a protocol doesn't work either:
from typing import *
type Fn[**P, R] = Callable[P, R]
class Decorator[**P, R](Protocol):
def __call__(self, fn: Fn[P, R], /) -> Fn[P, R]: ...
def do_the_thing[**P, R]() -> Decorator[P, R]:
def decorator(fn: Fn[P, R]) -> Fn[P, R]:
return fn
return decorator
@do_the_thing()
def blahblah() -> None:
pass
...so it seems the type information is just totally discarded.
Strangely, if I don't type-alias the decorator, then it works fine (which is a workaround, but very annoying and verbose for more complicated logic).
# this is fine.
def do_the_thing[**P, R]() -> Callable[[Fn[P, R]], Fn[P, R]]:
def decorator(fn: Fn[P, R]) -> Fn[P, R]:
return fn
return decorator
The problem is that I want to be able to alias Callable[[Fn[P, R]], Fn[P, R]] to a clean single identifier that I can reuse in the current type context, but I cannot find a nice way of doing that at the moment.
Expected Behavior
This should be valid.
Actual Behavior
main.py:14: error: Argument 1 has incompatible type "Callable[[], None]"; expected "Callable[[VarArg(Never), KwArg(Never)], Never]" [arg-type]
Found 1 error in 1 file (checked 1 source file)
Related Issues:
I came across GH-16512, but that was closed as fixed. Not sure if it is the same problem or not!
Your Environment
- Mypy version used: 1.14.1, 1.15.0
- Mypy command-line flags: none
- Mypy configuration options from
mypy.ini(and other config files): none - Python version used: 3.12, 3.13
- Reproducible in https://mypy-play.net/?mypy=1.15.0&python=3.13
I'm not sure whether this is a bug or intended behavior. Let's look at the decorator:
@deco()
def fn() -> None: ...
# is the same as
deco_callable = deco() # We don't know the type here - neither P nor R appeared so far!
def fn() -> None: ...
fn = deco_callable(fn) # So we fail here - P and R should've already been bound, but there is not enough info beforehand
Your Protocol approach is almost there. If you move the type vars to the appropriate scope (only decorator is semantically generic, not Decorator), it works (playground):
from typing import *
type Fn[**P, R] = Callable[P, R]
class Decorator(Protocol):
def __call__[**P, R](self, fn: Fn[P, R], /) -> Fn[P, R]: ...
def do_the_thing() -> Decorator:
def decorator[**P, R](fn: Fn[P, R]) -> Fn[P, R]:
return fn
return decorator
@do_the_thing()
def blahblah(x: str) -> None:
pass
reveal_type(blahblah) # N: Revealed type is "def (x: builtins.str)"
It isn't correct to say that a Decorator instance is generic - it can be applied to any callable, not to some specific one.
I'm not sure how to express that without protocol, unfortunately - let's wait for someone else to comment on that? There doesn't seem to be any way to propagate generics like you want, you need
type Decorator = Callable[[Fn[P, R]], Fn[P, R]]
so that it isn't generic itself, but binds typevars internally. I doubt it can be expressed without an explicit Protocol.
I'm not sure whether this is a bug or intended behavior.
Looking at this from the perspective of the typing spec, I think the code sample in the original repro should type check without error. As the OP notes, it type checks fine if you manually expand the type alias. Using a type alias shouldn't change the behavior. For comparison, it type checks fine in pyright.
perspective of the typing spec
Do I understand correctly that the spec requires that either both versions pass or they are both rejected, only mandating consistency? I can't find anything about such typevar propagation in Callable there.
In other words, according to the spec, does def fn[T]() -> Callable[[Callable[..., T]], Callable[..., T]] bind T early? Is fn() a Callable[[Callable[..., Unknown]], Callable[..., Unknown]] or a Callable[[Callable[..., T]], Callable[..., T]]?
And how about def fn[T](x: T) -> Callable[[Callable[..., T]], Callable[..., T]] - is the return type generic or bound to T from fn invocation?
Yeah, typevar binding for Callable is underspecified in the typing spec. (That's probably something we should fix.) I know that mypy and pyright both special-case the Callable special form when it is used within a return type annotation for a generic function.
The typing spec is less ambiguous (although admittedly not as clear as it could be) regarding type aliases. A type alias substitution shouldn't change the behavior. I think that means both versions should either pass or fail.
Yes, I agree that aliases should be treated in the same way, this part is indeed a bug - snippets with and without alias should be equivalent.
I learned only recently that the version without an alias works at all - IMO it's just wrong or at least ill-defined, while the explicit Protocol is an unambiguous solution. Are there any cases where a long-ish Callable solves a problem that a separate Protocol with explicit generics scope can't? (I never encountered one, but might be missing something)
Are there any cases where a long-ish
Callablesolves a problem that a separateProtocolwith explicit generics scope can't?
I can't think of any off the top of my head. However, there's unfortunately quite a bit of code that relies on the special-case behavior that was initially introduced by mypy (presumably before Protocol was even introduced to the type system). I replicated this in pyright for compatibility reasons even though I would have preferred not to. If we were to clean this up in the typing spec and mandate the use of Protocol for deferred type var binding, it would create pain for some developers. Maybe it's still worth it, but we'd probably need a lengthy deprecation plan.
Are there any cases where a long-ish Callable solves a problem that a separate Protocol with explicit generics scope can't?
Protocols in their current form require explicitly importing typing and have runtime evaluation costs, type alias statements are deferred and can be written in a way that the expression is never evaluated unless introspected.
Outside of that, they're also a lot more verbose without a reasonable reason, especially in the case of paramspec based callable declarations:
# Use below doesn't accept non-task Futures, so can't accept general awaitables
type CoroFunc[**P, R] = Callable[P, Coroutine[t.Any, t.Any, R]]
type TaskFunc[**P, R] = Callable[P, asyncio.Task[R]]
type TaskCoroFunc[**P, R] = CoroFunc[P, R] | TaskFunc[P, R]
type Deco[**P, R] = Callable[[TaskCoroFunc[P, R]], TaskFunc[P, R]]
def taskcache[**P, R](
ttl: float | None = None,
*,
cache_transform: CacheTransformer | None = None,
) -> Deco[P, R]:
The equivalent written with protocols and without type aliases doesn't have a meaningful difference:
class CoroFunc[**P, R](Protocol):
def __call__(self, *args: P.args, **kwargs: P.kwargs) -> Coroutine[t.Any, t.Any, R]:
...
class TaskFunc[**P, R](Protocol):
def __call__(self, *args: P.args, **kwargs: P.kwargs) -> asyncio.Task[R]:
...
class Deco[**P, R](Protocol):
def __call__(self, f: CoroFunc[P, R] | TaskFunc[P, R], /) -> TaskFunc[P, R]:
...
def taskcache[**P, R](
ttl: float | None = None,
*,
cache_transform: CacheTransformer | None = None,
) -> Deco[P, R]:
(I've copy pasted this from an existing codebase, CacheTransformer's declaration is left out, but irrelevant here other than that it would show even more verbosity)
I don't see any reason the examples here should fail, or why we should force this to require protocols
@mikeshardmind does that mean this is likely definitely a bug in MyPy as opposed to an issue with how I am approaching this?
@ascopes I consider this a bug in mypy. As Eric pointed out above:
A type alias substitution shouldn't change the behavior. I think that means both versions should either pass or fail.
If you can think of a reason why the two of my examples should not be seen as equivalent to a type checker, or why the latter should fail, then there's a potential argument that there's specification clarity at play, however I don't believe Paramspecs are unclear about their use in Callables or in Protocols currently.
We failed to expand the alias somewhere - yes, it is a bug, aliases (at least non-recursive) and their expansions should be completely equivalent.
But also we aren't required to support pushing unsolvable type variables to the Callable in the return type (even though it's currently implemented). The spec does not mention this scenario, and my common sense does not require it either (I even think it should not work this way, but it's been this way for a long time, so I agree with Eric that it would be difficult to axe entirely).
Btw, #18877 answers my question about "something doable with Callable but not with Protocol". Callables are assumed to undergo method binding, while Protocols are not. Mypy and pyright agree on that now.
When annotating a universal decorator factory, you need to make the inner decorator polymorphic instead of the outer decorator factory.
The OP can be solved with the following type hints:
from typing import *
type Fn[**P, R] = Callable[P, R]
class Decorator(Protocol): # polymorphic callback, so no T-Vars bound to the class
def __call__[**P, T](self, fn: Fn[P, T], /) -> Fn[P, T]: ...
def do_the_thing() -> Decorator:
def decorator[**P, R](fn: Fn[P, R]) -> Fn[P, R]:
return fn
return decorator
@do_the_thing()
def blahblah() -> None:
pass
https://mypy-play.net/?mypy=latest&python=3.12&gist=48c73f4d5fd987e4781dc669c3a24343
See Also: https://discuss.python.org/t/typing-a-decorator-factory/95850/3
In fact, since in this example the signature is identical, one can even simplify the ParamSpec usage away completely
from typing import *
class Decorator(Protocol): # polymorphic callback, so no T-Vars bound to the class
def __call__[Fn: Callable](self, fn: Fn, /) -> Fn: ...
def do_the_thing() -> Decorator:
def decorator[Fn: Callable](fn: Fn) -> Fn:
return fn
return decorator
@do_the_thing()
def blahblah(x: int, *, y: str) -> None:
pass
reveal_type(blahblah) # "def (x: builtins.int, *, y: builtins.str)"
https://mypy-play.net/?mypy=latest&python=3.12&gist=d5e796ffdbf323a809bdba484d53c3ae