python
Refactor all member access to go through checkmember.py (meta issue)
There are bunch of places where we duplicate some parts of code for special logic in member access (such as properties, class methods, descriptors, __getattr__(), etc):
- The
checkmember.pyitself - Override checks for variables
- Override checks for methods
- Multiple inheritance checks
- Checks for special methods via
check_op()andhas_member() - Protocol implementation check
Here are some issues that will be solved (or significantly simplified) by refactoring this to go through the same place (probably checkmember.py):
- https://github.com/python/mypy/issues/4125
- https://github.com/python/mypy/issues/5425
- https://github.com/python/mypy/issues/5136
- https://github.com/python/mypy/issues/3832
- https://github.com/python/mypy/issues/7565
- https://github.com/python/mypy/issues/7153
- https://github.com/python/mypy/issues/5836
- https://github.com/python/mypy/issues/5803
- https://github.com/python/mypy/issues/5491
- https://github.com/python/mypy/issues/5481
Also I think having member access cleaned-up, centralized, and documented will open the way to finally solving https://github.com/python/mypy/issues/708 (currently the oldest high priority issue), and to having general support for descriptor protocol (for historical reasons, several things like properties are supported via some special-casing).
Plugin management is also really important here, because right now protocols do not support objects with custom get_attribute_hook / get_method_sig_hook items. Quick example:
# ex.py
class A(object):
attr: str # is set via plugin
def method(self) -> str: # is set via plugin
...
a = A()
reveal_type(a.attr)
# Revealed type is "builtins.float"
reveal_type(a.method())
# Revealed type is "builtins.int"
Plugin:
from mypy.plugin import Plugin
class MyPlugin(Plugin):
def get_attribute_hook(self, fullname):
if fullname == 'ex.A.attr':
def test(ctx):
return ctx.api.named_type('builtins.float')
return test
def get_method_signature_hook(self, fullname: str):
if fullname == 'ex.A.method':
def test(ctx):
return ctx.default_signature.copy_modified(
ret_type=ctx.api.named_type('builtins.int'),
)
return test
def plugin(version):
return MyPlugin
At the moment, everything works as expected. Now, let's define a protocol and check if it works.
Protocol (problematic part):
from typing_extensions import Protocol
class Some(Protocol):
attr: float
def method(self) -> int:
...
def accepts_protocol(some: Some) -> int:
return some.method()
accepts_protocol(a)
# ex.py:26: error: Argument 1 to "accepts_protocol" has incompatible type "A"; expected "Some"
# ex.py:26: note: Following member(s) of "A" have conflicts:
# ex.py:26: note: attr: expected "float", got "str"
# ex.py:26: note: Expected:
# ex.py:26: note: def method(self) -> int
# ex.py:26: note: Got:
# ex.py:26: note: def method(self) -> str
find_member does not call get_attribute_hook because it does not even have chk instance.
At the moment I am trying to refactor find_member to be almost an alias to analyze_member_access. It is not so-commonly used. grep:
» ag find_member
mypy/checkmember.py
139: # TODO: This and following functions share some logic with subtypes.find_member;
mypy/join.py
15: is_protocol_implementation, find_member
592: return find_member('__call__', t, t, is_operator=True)
mypy/messages.py
38: is_subtype, find_member, get_member_flags,
599: call = find_member('__call__', original_caller_type, original_caller_type,
1922: if not find_member(member, left, left):
1936: supertype = find_member(member, right, left)
1938: subtype = find_member(member, left, left)
1957: if find_member(member, left, left):
mypy/constraints.py
324: call = mypy.subtypes.find_member('__call__', template, actual,
435: inst = mypy.subtypes.find_member(member, instance, subtype)
436: temp = mypy.subtypes.find_member(member, template, subtype)
481: call = mypy.subtypes.find_member('__call__', self.actual, self.actual,
mypy/checker.py
61: unify_generic_callable, find_member
4678: call = find_member('__call__', subtype, subtype, is_operator=True)
4683: call = find_member('__call__', supertype, subtype, is_operator=True)
4963: iter_type = get_proper_type(find_member('__iter__', instance, instance, is_operator=True))
mypy/subtypes.py
292: call = find_member('__call__', left, left, is_operator=True)
321: call = find_member('__call__', right, left, is_operator=True)
408: call = find_member('__call__', right, left, is_operator=True)
560: supertype = get_proper_type(find_member(member, right, left))
562: subtype = get_proper_type(find_member(member, left, left))
608:def find_member(name: str,
736: typ = get_proper_type(find_member(member, instance, instance))
1299: call = find_member('__call__', left, left, is_operator=True)
All things in checker.py are easy, I will just change find_member to analyze_member_access.
But I have several questions:
- What to do with functions that use
find_memberinside? Like ones insubtypes.pyandconstrains.py? They will require a lot of context to work properly:chkis the most complex one to pass through - How will it affect performance if
is_subtype(which is widely used) will callget_attribute_hook? - Should
find_memebrandanalyze_member_accessbe different? Are there any details to keep in mind?
Another problem that came to my mind while thinking about this is how to handle deferrals? If e.g. is_subtype() ultimately ends up in protocol check w.r.t. to a class with not ready attribute (e.g. a tricky decorator) then we can't really return neither False nor True since both can cause spurious errors. The only way I see is to raise an error, but then each call to is_subtype() should expect this, making everything insanely complex.
I think this issue can now be closed, we are in a much better state now IMO. I opened https://github.com/python/mypy/issues/19299 for some followup tasks and ideas (I may work om some of them soon depending on other priorities).