__dict__ attribute of a meta-class is incorrectly typed.

[tim-mitchell]

Initially lodged as https://github.com/python/mypy/issues/16501

Given a metaclass called Meta, mypy considers the Meta.dict attribute to have type def (self: builtins.type) -> types.MappingProxyType[builtins.str, Any] when the actual type is types.MappingProxyType[builtins.str, Any].

To Reproduce Run mypy on this file.

class Meta(type):
    def __call__(cls, *args, **kw):
        return super().__call__(*args, **kw)

reveal_type(Meta.__dict__)
call = Meta.__dict__['__call__']  # mypy raises error here

https://mypy-play.net/?mypy=latest&python=3.11&gist=843cafceeef590efc6d3705b44c29c19

Expected Behavior

reveal_type(Meta.__dict__) returns types.MappingProxyType[builtins.str, Any]

Actual Behavior

mypy_test.py:6: note: Revealed type is "def (self: builtins.type) -> types.MappingProxyType[builtins.str, Any]"

This is likely due to builtins.pyi defining __dict__ as a property.

class type:
    ...
    @property
    def __dict__(self) -> types.MappingProxyType[str, Any]: ...  # type: ignore[override]

[AlexWaygood]

In the mypy issue you originally opened, @erictraut wrote:

I think mypy is doing the correct thing here given the way dict is defined in builtins.pyi. It's defined as a property, and an access to a property through a class returns the property object.

class type:
    ...
    @property
    def __dict__(self) -> types.MappingProxyType[str, Any]: ...  # type: ignore[override]

FWIW, pyright's behavior is the same as mypy's.

I suspect that dict was modeled as a property in typeshed because it is read-only. If the type system were to include support for a ReadOnly qualifier like the one proposed in draft PEP 705, typeshed could model this better.

I disagree with Eric's analysis in some respects.

Firstly, is type.__dict__ a property? No, not strictly speaking. However, it is a "non-data descriptor" (a descriptor that defines __get__, but not __set__. In this respect, it's very much analogous to properties that don't define setters, only getters. For nearly all "property-like" descriptors in typeshed, we approximate the type using @property, since (1) it's an easy shorthand for us, and (2) historically, type checkers have implemented a lot of special-casing for @property which means that they handle the behaviour of @property much better than with other descriptors.

Here's what I mean when I say that type.__dict__ is actually a descriptor:

>>> type.__dict__
mappingproxy({'__new__': <built-in method __new__ of type object at 0x00007FFD4B4713D0>, '__repr__': <slot wrapper '__repr__' of 'type' objects>, '__call__': <slot wrapper '__call__' of 'type' objects>, '__getattribute__': <slot wrapper '__getattribute__' of 'type' objects>, '__setattr__': <slot wrapper '__setattr__' of 'type' objects>, '__delattr__': <slot wrapper '__delattr__' of 'type' objects>, '__init__': <slot wrapper '__init__' of 'type' objects>, '__or__': <slot wrapper '__or__' of 'type' objects>, '__ror__': <slot wrapper '__ror__' of 'type' objects>, 'mro': <method 'mro' of 'type' objects>, '__subclasses__': <method '__subclasses__' of 'type' objects>, '__prepare__': <method '__prepare__' of 'type' objects>, '__instancecheck__': <method '__instancecheck__' of 'type' objects>, '__subclasscheck__': <method '__subclasscheck__' of 'type' objects>, '__dir__': <method '__dir__' of 'type' objects>, '__sizeof__': <method '__sizeof__' of 'type' objects>, '__basicsize__': <member '__basicsize__' of 'type' objects>, '__itemsize__': <member '__itemsize__' of 'type' objects>, '__flags__': <member '__flags__' of 'type' objects>, '__weakrefoffset__': <member '__weakrefoffset__' of 'type' objects>, '__base__': <member '__base__' of 'type' objects>, '__dictoffset__': <member '__dictoffset__' of 'type' objects>, '__name__': <attribute '__name__' of 'type' objects>, '__qualname__': <attribute '__qualname__' of 'type' objects>, '__bases__': <attribute '__bases__' of 'type' objects>, '__mro__': <attribute '__mro__' of 'type' objects>, '__module__': <attribute '__module__' of 'type' objects>, '__abstractmethods__': <attribute '__abstractmethods__' of 'type' objects>, '__dict__': <attribute '__dict__' of 'type' objects>, '__doc__': <attribute '__doc__' of 'type' objects>, '__text_signature__': <attribute '__text_signature__' of 'type' objects>, '__annotations__': <attribute '__annotations__' of 'type' objects>, '__type_params__': <attribute '__type_params__' of 'type' objects>})
>>> type(type.__dict__)
<class 'mappingproxy'>
>>> type.__dict__['__dict__']
<attribute '__dict__' of 'type' objects>
>>> type(_)
<class 'getset_descriptor'>
>>> type.__dict__['__dict__'].__get__(type)
mappingproxy({'__new__': <built-in method __new__ of type object at 0x00007FFD4B4713D0>, '__repr__': <slot wrapper '__repr__' of 'type' objects>, '__call__': <slot wrapper '__call__' of 'type' objects>, '__getattribute__': <slot wrapper '__getattribute__' of 'type' objects>, '__setattr__': <slot wrapper '__setattr__' of 'type' objects>, '__delattr__': <slot wrapper '__delattr__' of 'type' objects>, '__init__': <slot wrapper '__init__' of 'type' objects>, '__or__': <slot wrapper '__or__' of 'type' objects>, '__ror__': <slot wrapper '__ror__' of 'type' objects>, 'mro': <method 'mro' of 'type' objects>, '__subclasses__': <method '__subclasses__' of 'type' objects>, '__prepare__': <method '__prepare__' of 'type' objects>, '__instancecheck__': <method '__instancecheck__' of 'type' objects>, '__subclasscheck__': <method '__subclasscheck__' of 'type' objects>, '__dir__': <method '__dir__' of 'type' objects>, '__sizeof__': <method '__sizeof__' of 'type' objects>, '__basicsize__': <member '__basicsize__' of 'type' objects>, '__itemsize__': <member '__itemsize__' of 'type' objects>, '__flags__': <member '__flags__' of 'type' objects>, '__weakrefoffset__': <member '__weakrefoffset__' of 'type' objects>, '__base__': <member '__base__' of 'type' objects>, '__dictoffset__': <member '__dictoffset__' of 'type' objects>, '__name__': <attribute '__name__' of 'type' objects>, '__qualname__': <attribute '__qualname__' of 'type' objects>, '__bases__': <attribute '__bases__' of 'type' objects>, '__mro__': <attribute '__mro__' of 'type' objects>, '__module__': <attribute '__module__' of 'type' objects>, '__abstractmethods__': <attribute '__abstractmethods__' of 'type' objects>, '__dict__': <attribute '__dict__' of 'type' objects>, '__doc__': <attribute '__doc__' of 'type' objects>, '__text_signature__': <attribute '__text_signature__' of 'type' objects>, '__annotations__': <attribute '__annotations__' of 'type' objects>, '__type_params__': <attribute '__type_params__' of 'type' objects>})

So, how do we square that with Eric's comments that "an access to a property through a class returns the property object"? Well, here we get to the unique nature of type as a class:

>>> type(type) is type
True

type is the class of instances of type such as int, so from that perspective, you'd think that type.__dict__ should give you the descriptor itself. But type is also an instance of type, so from that perspective, it's logical that accessing the __dict__ attribute on type would call type.__dict__['__dict__'].__get__ in the same way that accessing any other property on an instance of a class would call __get__ on that property.

Where does this leave us? Well, this is truly an edge-case in Python's data model and in the type system! I don't think typeshed's annotations are incorrect in the exact way Eric suggests, and I think the only way of getting the semantics accurate in all edge cases would be for mypy and pyright to implement some special-casing here. However, I'd certainly be open to changing typeshed's annotations here if somebody has a concrete suggestion for how to improve the situation. (We don't have a ReadOnly type yet, so that's not an option for now.)

@erictraut, curious for your thoughts here :)

[tim-mitchell]

It seems to me that the type is also an instance of type perspective matches the actual run-time behaviour. But I'll leave it to you two to sort out. :)

[erictraut]

@AlexWaygood, I agree with your observation that type is an instance of type, but it's also a subclass of object. The object class already provides a type declaration for the variable __dict__. I think you're arguing that type checkers should ignore the type.__dict__ override and use the object.__dict__ definition in some cases. I'm not convinced such special-casing is required.

Let's first explore why type.__dict__ overrides object.__dict__. I presume there are two reasons. The first is because type.__dict__ is read-only at runtime, but __dict__ is not read-only for most other subclasses of object.

class Foo: ...
Foo().__dict__ = {} # Not a problem

Foo.__dict__ = {} # "AttributeError: attribute '__dict__' of 'type' objects is not writable"

The second (and probably more important) reason is that the dictionary itself is read-only. It's actually not an instance of dict but rather an instance of MappingProxyType, which is effectively a read-only dict.

Foo().__dict__["bar"] = "" # Not a problem

Foo.__dict__["bar"] = "" # "TypeError: 'mappingproxy' object does not support item assignment"

The problem with the current typeshed definition for type.__dict__ is that it overrides the definition of object.__dict__ in a way that is not faithful to the runtime. At runtime, type.__dict__ is not a descriptor (property or otherwise). It's simply a class variable that contains an instance of mappingproxy. If typeshed modeled it accordingly, pyright (and presumably mypy) would handle this case correctly.

class type:
    ...
    __dict__: ClassVar[types.MappingProxyType[str, Any]]

Do you see a problem with making this change in typeshed? The one downside I see is that assignments to type.__dict__ will no longer be flagged as violations because it won't be considered read-only, but it's unlikely that someone's going to try to construct a new instance of MappingProxyType and assign it to type.__dict__ (type.__dict__ = MappingProxyType({})).

If you're concerned about this, we could use Final instead of ClassVar. That would make it read-only as well as a ClassVar.

class type:
    ...
    __dict__: Final[types.MappingProxyType[str, Any]]

I've verified that with this change, pyright no longer reports a type violation for the code at the start of this thread.

[Daverball]

While custom descriptors are not quite as well supported as property I still think you might be able to get away with one, since this is a much more simple case than your average descriptor:

class _ReadOnly(Protocol(_T_co)):
    def __get__(self, __obj: object, __owner: type[object] | None = None) -> _T_co: ...

class type:
    ...
    __dict__: _ReadOnly[types.MappingProxyType[str, Any]]

Without the usual overload for __obj: None you will always get back the MappingProxyType. And assignments are still disallowed since the descriptor has no __set__/__delete__.

That being said I think the solution with Final/ClassVar is perfectly adequate as well.