twisted
Better Deferred type hints for callbacks that return `_T|Deferred[_T]`
It looks like there are some cases where a callback that returns a type or a Deferred of the same type is not accepted by mypy.
The most basic case, which works fine:
from twisted.internet.defer import Deferred
def cb(result: int) -> Deferred[int] | int:
...
d: Deferred[int] = Deferred()
d.addCallback(cb)
This one doesn't:
from twisted.internet.defer import Deferred
from twisted.python.failure import Failure
def cb(result: int | Failure) -> Deferred[int] | int | Failure:
...
d: Deferred[int] = Deferred()
d2: Deferred[int] = d.addBoth(cb)
"Argument 1 to "addBoth" of "Deferred" has incompatible type "Callable[[int | Failure], Deferred[int] | int | Failure]"; expected "Callable[[int | Failure], Failure]" [arg-type]"
In this second snippet removing any detail of it (removing the d2 assignment or its type hint, removing either int or Deferred[int] from the callback return type, using addCallback instead of addBoth + removing Failure from the arg and the return value) removes the error, but simply replacing addBoth with addCallback doesn't. Removing the annotation from d2 and running reveal_type on it gives twisted.internet.defer.Deferred[builtins.object].
Another unrelated example:
from twisted.internet.defer import Deferred, maybeDeferred
def fn() -> int | Deferred[int]:
...
d: Deferred[int] = maybeDeferred(fn)
"Argument 1 to "maybeDeferred" has incompatible type "Callable[[], int | Deferred[int]]"; expected "Callable[[], Deferred[int]]" [arg-type]"
Here removing the annotation from d and running reveal_type on it gives twisted.internet.defer.Deferred[Union[builtins.int, twisted.internet.defer.Deferred[builtins.int]]].
This is definitely a thing that happens, but I am not seeing how it is actionable — is there something twisted could change here?
I just assumed it's something that is missing in the Twisted type hints. As overloads for both addCallback/addBoth and maybeDeferred are quite complex it's not immediately obvious whether they support this or not. Do you think if it's possible to make sure Twisted is correct in this regard, by looking closer at its type hints?
I just assumed it's something that is missing in the Twisted type hints. As overloads for both
addCallback/addBothandmaybeDeferredare quite complex it's not immediately obvious whether they support this or not. Do you think if it's possible to make sure Twisted is correct in this regard, by looking closer at its type hints?
Having looked quite closely at those type hints when I last rewrote them, I believe that this is about as well as we can do. There are some limitations in Mypy which make it impossible to express some of the constraints. I would love for someone to point out a way to be more precise though, so I will leave this open if you're inclined to give it a shot yourself.
An improvement here would be to separate what is logically the Functor part of addCallback (callbacks returning T - aka fmap) from what is logically the Monad part of addCallback (returning Deferred[T] - aka bind).
I expect any such transition would be quite long but could have a number of benefits in terms of code complexity (eliminating it) and performance (improving it).
