typing
typing copied to clipboard
python
Higher-Kinded TypeVars
aka type constructors, generic TypeVars
Has there already been discussion about those? I do a lot of FP that results in impossible situations because of this. Consider an example:
A = TypeVar('A')
B = TypeVar('B')
F = TypeVar('F')
class M(Generic[F[X], A]):
def x(fa: F[A], f: Callable[[A], B]) -> F[B]:
return map(f, fa)
M().x([1], str)
I haven't found a way to make this work, does anyone know a trick or is it impossible?
If not, consider the syntax as a proposal.
Reference implementations would be Haskell, Scala.
optimally, the HK's type param would be indexable as well, allowing for F[X[X, X], X[X]]
Summary of current status (by @smheidrich, 2024-02-08):
- @JelleZijlstra has indicated interest in sponsoring a PEP, conditional on a prototype implementation in a major type checker and a well-specified draft PEP.
- Drafting the PEP takes place in @nekitdev's fork of the
pepsrepo. The stub PEP draft so far contains a few examples of the proposed syntax. - That same repo's GitHub Discussions forum forum has been designated as the place to discuss the PEP (and presumably the prototype implementation?). Some limited further discussions have taken place there.
- If you want to be notified of new discussion threads, I think you have to set the whole repo as "watched" in GitHub?
I think this came up few times in other discussions, for example one use case is https://github.com/python/mypy/issues/4395. But TBH this is low priority, since such use cases are quite rare.
damn, I searched very thoroughly but did not find this one! :smile: So, consider this my +1!
Adding your +1 is all fine, but who's going to do the (probably pretty complicated) implementation work?
I am neither approving nor disapproving. Just observing that it may be a lot of work for marginal benefits in most codebases.
awfully pragmatic. Where's your sense of adventure? :smile: anyways, I'll work on it, though it's gonna take a while to get into the project.
Similar to https://github.com/Microsoft/TypeScript/issues/1213
Not sure if the discussion over there provides any useful insights to the effort over here.
Hi @tek. I'm also very interested in this, so I'd like to ask if you had any progress with this and volunteer to help if you want.
@rcalsaverini sorry, I've been migrating my legacy code to haskell and am abandoning python altogether. but I wish you great success!
Just to add another use case (which I think relates to this issue):
Using Literal types along with overloading __new__, along with higher-kinded typevars could allow implementing a generic "nullable" ORM Field class, using a descriptor to provide access to the appropriate nullable-or-not field values. The descriptor wouldn't have to be reimplemented in subclasses.
It is one step closer to being possible due to the most recent mypy release's support for honoring the return type of __new__ (https://github.com/python/mypy/issues/1020).
Note: this is basically a stripped-down version of Django's Field class:
# in stub file
from typing import Generic, Optional, TypeVar, Union, overload, Type
from typing_extensions import Literal
_T = TypeVar("_T", bound="Field")
_GT = TypeVar("_GT")
class Field(Generic[_GT]):
# on the line after the overload: error: Type variable "_T" used with arguments
@overload
def __new__(cls: Type[_T], null: Literal[False] = False, *args, **kwargs) -> _T[_GT]: ...
@overload
def __new__(cls: Type[_T], null: Literal[True], *args, **kwargs) -> _T[Optional[_GT]]: ...
def __get__(self, instance, owner) -> _GT: ...
class CharField(Field[str]): ...
class IntegerField(Field[int]): ...
# etc...
# in code
class User:
f1 = CharField(null=False)
f2 = CharField(null=True)
reveal_type(User().f1) # Expected: str
reveal_type(User().f2) # Expected: Union[str, None]
I wonder if this is what I need or if there's currently a work around for my (slightly simpler) case?:
I'm building an async redis client with proper type hints. I have a "Commands" class with methods for all redis commands (get, set, exists, strlen ... and hundreds more). Normally each of those methods should return a future (actually coroutine) to the result, but in pipeline mode they should all return None - the commands are added to the pipeline to be executed later.
This is easy enough to implement in python, but not so easy to type hint correctly.
Basic example:
class Redis:
def execute(self, command) -> Coroutine[Any, Any, Union[None, str, int, float]]:
return self.connection.execute(...)
def get(self, *args) -> Coroutine[Any, Any, str]:
...
return self.execute(command)
def set(self, *args) -> Coroutine[Any, Any, None]:
...
return self.execute(command)
def exists(self, *args) -> Coroutine[Any, Any, bool]:
...
return self.execute(command)
# ... and many MANY more ...
class RedisPipeline(Redis):
def execute(self, command) -> None:
self.pipeline.append(command)
I tried numerous options to make Coroutine[Any, Any, xxx] generic, but nothing seems to work.
Is there any way around this with python 3.8 and latest mypy? If not a solution would be wonderful - as far as I can think, my only other route for proper types is a script which copy and pastes the entire class and changes the return types in code.
@samuelcolvin I don't think this question belongs in this issue. The reason for the failure (knowing nothing about Redis but going purely by the code you posted) is that in order to make this work, the base class needs to switch to an Optional result, i.e.
def execute(self, command) -> Optional[Coroutine[Any, Any, Union[None, str, int, float]]]:
I get that, but I need all the public methods to definitely return a coroutine. Otherwise, if it returned an optional coroutine, it would be extremely annoying to use.
What I'm trying to do is modify the return type of many methods on the sub-classes, including "higher kind" types which are parameterised.
Hence thinking it related to this issue.
Honestly I have no idea what higher-kinded type vars are -- my eyes glaze over when I hear that kind of talk. :-)
I have one more suggestion, then you're on your own. Use a common base class that has an Optional[Coroutine[...]] return type and derive both the regular Redis class and the RedisPipeline class from it.
Okay, so the simple answer is that what I'm trying to do isn't possible with python types right now.
Thanks for helping - at least I can stop my search.
I suspect that the reason is that it simply isn't type-safe, and you couldn't do it (using subclassing) in any other typed language either.
humm, but the example above under "Basic example" I would argue IS type-safe.
All the methods which end return self.execute(...) return what execute returns - either a Coroutine or None.
Thus I don't see how this as any more "unsafe" than normal use of generics.
@gvanrossum, I can relate!
I wonder if bidict provides a practical example of how this issue prevents expressing a type that you can actually imagine yourself needing.
>>> element_by_atomicnum = bidict({0: "hydrogen", 1: "helium"})
>>> reveal_type(element_by_atomicnum) # bidict[int, str]
# So far so good, but now consider the inverse:
>>> element_by_atomicnum.inverse
bidict({"hydrogen": 0, "helium": 1})
What we want is for mypy to know this:
>>> reveal_type(element_by_atomicnum.inverse) # bidict[str, int]
merely from a type hint that we could add to a super class. It would parameterize not just the key type and the value type, but also the self type. In other words, something like:
KT = TypeVar('KT')
VT = TypeVar('VT')
class BidirectionalMapping(Mapping[KT, VT]):
...
def inverse(self) -> $SELF_TYPE[VT, KT]:
...
where $SELF_TYPE would of course use some actually legal syntax that allowed composing the self type with the other parameterized types.
Okay, I think that example is helpful. I recreated it somewhat simpler (skipping the inheritance from Mapping and the property decorators):
from abc import abstractmethod
from typing import *
T = TypeVar('T')
KT = TypeVar('KT')
VT = TypeVar('VT')
class BidirectionalMapping(Generic[KT, VT]):
@abstractmethod
def inverse(self) -> BidirectionalMapping[VT, KT]:
...
class bidict(BidirectionalMapping[KT, VT]):
def __init__(self, key: KT, val: VT):
self.key = key
self.val = val
def inverse(self) -> bidict[VT, KT]:
return bidict(self.val, self.key)
b = bidict(3, "abc")
reveal_type(b) # bidict[int, str]
reveal_type(b.inverse()) # bidict[str, int]
This passes but IIUC you want the ABC to have a more powerful type. I guess here we might want to write it as
def inverse(self: T) -> T[VT, KT]: # E: Type variable "T" used with arguments
Have I got that?
Exactly! It should be possible to e.g. subclass bidict (without overriding inverse), and have mypy realize that calling inverse on the subclass gives an instance of the subclass (with the key and value types swapped as well).
This isn’t only hypothetically useful, it’d really be useful in practice for the various subclasses in the bidict library where this actually happens (frozenbidict, OrderedBidict, etc.).
Glad this example was helpful! Please let me know if there’s anything further I can do to help here, and (can’t help myself) thanks for creating Python, it’s such a joy to use.
Ah, so the @abstractmethod is also a red herring.
And now I finally get the connection with the comment that started this issue.
But I still don't get the connection with @samuelcolvin's RedisPipeline class. :-(
I would also say that this example is really simple, common, but not supported:
def create(klass: Type[T], value: K) -> T[K]:
return klass(value)
We use quite a lot of similar constructs in dry-python/returns.
As a workaround I am trying to build a plugin with emulated HKT, just like in some other languages where support of it is limited. Like:
- Swift and
bow: https://bow-swift.io/docs/fp-concepts/higher-kinded-types/ - TypeScript and
fp-ts: https://github.com/gcanti/fp-ts/blob/master/src/HKT.ts
Paper on "Lightweight higher-kinded polymorphism": https://www.cl.cam.ac.uk/~jdy22/papers/lightweight-higher-kinded-polymorphism.pdf
TLDR: So, instead of writing T[K] we can emulate this by using HKT[T, K] where HKT is a basic generic instance processed by a custom mypy plugin. I am working on this plugin for already some time now, but there's still nothing to show. You can track the progress here: https://pypi.org/project/kinds/ (part of dry-python libraries)
But I still don't get the connection with @samuelcolvin's RedisPipeline class. :-(
Sorry if I wasn't clear. I'll try again to explain:
I have a class with many (~200) methods, they all return coroutines with different result types (None, bytes, str, int or float). I want a subclass with the same internal logic but where all those methods return None - I can do this in python, but not with type hints currently (here's the actual code if it helps)
So roughly I want:
T = TypeVar('T', bytes, str, int, float, 'None')
Result = Coroutine[Any, Any, T]
class Foo:
def method_1(self, *args) -> Result[str]:
...
def method_2(self, *args) -> Result[None]:
...
def method_3(self, *args) -> Result[bool]:
...
...
...
def method_200(self, *args) -> Result[int]:
...
class Bar:
def method_1(self, *args) -> None:
...
def method_2(self, *args) -> None:
...
def method_3(self, *args) -> None:
...
...
...
def method_200(self, *args) -> None:
...
Except I don't want to have to redefine all the methods on Bar. Assuming I could create my own AlwaysNone type which even when parameterised told mypy the result would always be None
class AlwaysNoneMeta(type):
def __getitem__(self, item) -> None:
return None
class AlwaysNone(metaclass=AlwaysNoneMeta):
pass
I think the feature requested here could solve my problem.
In other words if mypy could understand
def generate_cls(OuterType):
class TheClass:
def method_1(self, *args) -> OuterType[str]:
...
...
return TheClass
Foo = generate_cls(Result)
Bar = Generate_cls(AlwaysNone)
I'd be in the money, but quite understandably it can't.
I've currently got a working solution where i generate a .pyi stub file with definitions for all these methods but changing the return type to None (see here) so I'm not in immediate need of this anymore.
It looks like you want completely different signatures whose likeness is limited to their names and arguments:
async def do_work(redis_client: Redis):
await redit_sclient.get(...)
do_work(RedisPipeline()) # user expects type checker to warn about misuse
It's understandable you're looking to reuse some common "template", but I fail to see what it has to do with higher-kinded types.
Hi @Kentzo, that's incorrect, these are not completely different signatures.
Please review the code here; as you can see these are functions which may return an awaitable object, not coroutines. This approach and similar, while not necessarily obvious to beginners, is relatively common in libraries which make extensive use of asyncio and the standard library asyncio code.
If you're not sure how it works, feel free to submit an issue on that project and I'll endeavour to explain it to you.
In your initial example:
class Redis:
def execute(self, command) -> Coroutine[Any, Any, Union[None, str, int, float]]:
...
class RedisPipeline(Redis):
def execute(self, command) -> None:
These are different signatures. Functions annotated to accept Redis as an argument would expect execute to return a coroutine.
This is off-topic, please create an issue on async-redis if you want to discuss this more.
In short, yes they're different signatures, but I think my explanation above gives enough detail on what I'm doing and why I think it relates to this issue.
+1 on this. It's hard to say the use cases are rare when it's not even possible to express... There may be countless use cases hiding behind duck typing that simply can't be checked by the type system and therefore certainly aren't recognized as such. Having a type safe generic "Mappable" class as shown in the OP would be really useful.
It's hard to say the use cases are rare when it's not even possible to express
That makes no sense. You can express the OP's example just fine in untyped Python, you just can't express the types correctly. So if there were "countless" real examples we would encounter frequent user requests for type system extensions to allow typing this correctly. I haven't seen a lot of those.
Do you have an actual (non-toy) example?