micropython
typing: Add typing module
Brought up a couple of times here and on the forums.
To implement this I just took most things from dir(typing) in CPython 3.8, removed what looked too esoteric and kept the rest. Alphabetically so it's easier to find things. Probably still too much, but I really cannot guess what is used and what is not.
Closes #190.
There's also a pretty good / simple typing module here: https://github.com/pfalcon/pycopy-lib/tree/master/typing I've got a rebased copy of this locally that I pulled out as part of my (unfinished) pdb work if it looks useful?
I also want to add, thanks for starting this!
While there are a few stub / hack methods of working around typing enough to use autocomplete in IDE's, there's value in a more featured typing module for use with mypy and related tooling
looks useful
yes but seemed fairly old and had less types than I found in CPython so I just started from scratch
Would love this. I didn't even realize until just now that there isn't a typing module till I tried to add type hints for an optional parameter just now. Thanks you!
Thanks for the reminder @andrewleech. Some thoughts....
- Having a
typingmodule on the device is obviously important if you want to be able to directly run code that has type annotations. However, I understand that most of this is just to make thefrom typing import Listline work, because the actual annotations are ignored by the compiler (i.e. I can writedef foo(x: abc) -> xyz:) and the compiler doesn't care ifabcorxyzexist. - However, it's a pretty unfortunate waste of flash+RAM to just make a no-op import work.
So... could we get most of what people need by having typing.py just implement a module-level __getattr__ that returns None to every attribute? Then you could write from typing import AnythingYouLike and the compiler would be happy.
Furthermore, could we take this one step further and implement a special case in the compiler to just ignore any import statements from typing ?
Additionally, as far as I understand Python 3.9 now mostly makes importing stuff from the typing module obsolete. You can write x: list[int] now (not List[int]). I guess code will continue to exist for a long time with typing imports.
I like most of what's proposed here, in particular having a minimal on-board "stub" to reduce runtime costs.
Even with the newer syntax not needing eg. List there's still a bunch of typing only "constructs" like Union or Optional which I use a lot, though maybe there's new syntax to replace that too... I'll have to take a look.
After that there's an even shorter list of functions in typing such as NewType and cast which are used outside type annotations, so would crash out as a None.
Maybe instead of returning None the getattr could return something callable like
def null(*args, **kwargs):
return None
or a similar lambda or similar?
Furthermore, could we take this one step further and implement a special case in the compiler to just ignore any import statements from typing ?
I think this should suffice, and then this would be my preferred way: no typing module needed at all, all type annotations simply ignored.
I had a go at implementing a "simplified" on-device stub version based on a __getattr__ approach, immediately ran into a bunch of issues with my existing project codebase :-(
It's common practice when using type hinting to create type aliases of complex types for reuse, eg.
from typing import Union, Tuple
LedLevels: TypeAlias = Tuple[Union[int, float, None], Union[int, float, None], Union[int, float, None]]
While it might just be a case of "you can't do this with microypthon" it's a bit of a shame... it works with the existing stub library here.
This slightly bigger on-device stub does seem to work with my above use case and any similar "real" use of typing features:
class _TypeIgnore:
def __init__(*args, **kwargs):
pass
def __call__(self, *args, **kwargs):
return self
def __getitem__(self, attr):
return self
_type_ignore = _TypeIgnore()
def __getattr__(attr):
return _type_ignore
If the stub/mock works then what would still be the benefit of keeping the rest? Catching import errors like typos? Would also be a bit tedious to have to manually change the TYPE_CHECKING constant.
Yeah I'm a little conflicted about keeping the rest... having it there can help autocomplete / linting in some tools. You don't need to manually change the const, the tools generally do that dynamically themselves.
However I'm working on getting full mypy working with https://github.com/Josverl/micropython-stubs and the typing stub modules there, so this copy isn't actually used on PC in that case.
having it there can help autocomplete / linting in some tools
If you're linting, you're using CPython or similar anyway which has its own proper typing module (well, I guess, see question below). For autocomplete it might indeed help, on the other hand I'm not sure if it's really such an issue if it doesn't (especially if you're used writing micropython-based code already) since it's mostly on a 'first time it is needed' basis anyway: once you've added a couple of typing statements in a project any sane autocomplete will pick them up.
and the typing stub modules there
Why does it need a typing stub, can't it not use CPython's typing module? (with the benefit that a linter would then effectively interpret the types correctly, not sure if it can do that based on a stub?).
There's a few weird things about the real cpython typing module, for example it pulls in a few packages to create type aliases like importing collections.defaultdict to alias to DefaultDict.
If I'm trying to use micropython specific stubs for built-in packages there's a bunch of these that get broken.
If I let it use cpython from the dependent imports, it ends up being a long thread of various not-quite-compatible stuff that results in more and more errors.
So it ends up being a trimmed / modified typing module being used in the stubs package.
To be clear the typing module I'm taking about is part of the stdlib stubs package and is based on the real cpython one, it's not really a stub as such but is named pyi, it's intended to be consumed by tools rather than executed I guess.
https://pypi.org/project/micropython-stdlib-stubs
I've patched this more for mypy compat but haven't managed to figure out how to PR my patches upstream yet.
This doesn't really explain why I should keep any of the extra detail in this installable typing module... maybe there really isn't any reason at all.
@andrewleech lets chat,. I'm interested in what changes you made to get mypy working better. micropython-stdlib-stubs is maintained manually , and I've got quality tests setup that can be extended to mypy, so making changes should be simpler with little risk of degradation
@Josverl yes I've been meaning to reach out / raise a bunch of merge requests, but each time I think I've got it all working I find another exclusion / override left in my pyproject.toml that, once reset to defaults, highlights a bunch of things still broken.
My plan is to finish cleaning up basic support, then get changes into git, then slice and dice the changes into stdlib logical commits/pr and other things into the code generator stuff as appropriate.
It's just been a slow part time project so far, but has already unearthed some bugs / incorrect code in my application codebase so showing its value!
On the types needed by the stubs:
Currently the types I use to build the stubs are the Uppercase variants in order to be compatible with older Python versions. Ive not researched it yet, but as typing is more and more part of the newer Python versions, I think that could probably be relaxed to Types that are already available without adding overhead Below is the entire list currently explicitly used in micropython-stubs
TYPING_IMPORT: List[str] = [
"from typing import IO, Any, Callable, Coroutine, Dict, Generator, Iterator, List, NoReturn, Optional, Tuple, Union, NamedTuple, TypeVar",
"from _typeshed import Incomplete",
]
a few likely candidates for reduction are:
- Dict --> dict
- List --> list
- Iterator --> list
- Tuple --> tuple
- NamedTuple --> collections.namedtuple
Micropython-stdlib has a bunch more - but perhaps that could be also trimmed down further.
Would that help ?
@andrewleech ,
Ive been bundling the typings.py in micropython-stubs , and one of the users, @ANogin, found a runtime issue when defining Generics.
The error reported is on class _AnyCall ,
TypeError: '_AnyCall' object isn't subscriptablehttps://github.com/Josverl/micropython-stubs/issues/755
I cannot find a reference of _AnyCall in pylance/mypy typing (Python 3.11), while I can find usage of _AnyCallable.
_AnyCallable: TypeAlias = Callable[..., object]infunctools.pyi
what was the origin of your _AnyCall and could / should we adopt it or replace it by _AnyCallable ?
Is that the same / similar ?
The AnyCall in this PR doesn't come from anywhere necessarily I don't think, it's just a minimal stub that implements __call__().
It's then wrapped in simple Subscriptable type that's used by everything that should be used the way, it would be useful to see a bug report about what usage threw the error?
The AnyCall in this PR doesn't come from anywhere necessarily I don't think, it's just a minimal stub that implements
__call__().It's then wrapped in simple Subscriptable type that's used by everything that should be used the way, it would be useful to see a big report about what usage threw the error?
See https://github.com/Josverl/micropython-stubs/issues/755#issuecomment-2136479507 for a specific example of the error.
is this still something being considered, or is typing going to be ignored by the interpreter?
