Typing: include field type in Attribute type signature

[Tinche]

I'm already using attrs to implement several internal ORMs, and attrs (with cattrs) is very good for this. However I find myself missing a feature.

Consider the following code:

import attr


@attr.define
class A:
    a: int


reveal_type(attr.fields(A).a)

The revealed type is attr.Attribute[Any]. What would it take for mypy to think the type was attr.Attribute[int]?

This would be hugely useful for enabling type-safe projections. Often times, whether you're using a relational database or a key-value store like Mongo, you're only interested in a subset of the data. It's natural to represent your database model as a fully type-annotated attrs class, and it would be great to be able to implement something like:

from bson import ObjectId  # Pretend we're working with Mongo

async def load_projection(id: ObjectId, field: attr.Attribute[T]) -> T:
    pass

my_a = await load_projection(id, fields(A).a)

And have mypy properly check it.

[hynek]

That would be a thing for the mypy plugin, no? Maybe it would be possible if we follow through with #421?

[Tinche]

That would be a thing for the mypy plugin, no?

Yes probably, did I file this in the wrong place? I'm not sure where the mypy plugin is developed.

[hynek]

Nobody is. 🤪

They merged @euresti's PR but I think we should still take it over. Unfortunately the "we" doesn't include me, because this is all way above my head.

[Tinche]

This is above my head too but the potential benefits are too great, I guess I'll have to start reading 😇

[euresti]

Hi. Yes this is definitely something a Plugin would have to do. I've been trying to find some free time to move the plugin over here. But I still need to figure out how to move the tests over. (Our current testing of the stubs doesn't cover error it only tests that code doesn't fail IIRC)

[Tinche]

Just so we're clear, I want to express my continued appreciation for your work on this @euresti . If we can get this done I think attrs could basically become indispensible when dealing with typed Python.

[Tinche]

Has there been any movement on this? If we can get this working, we would be the basis for the first Python orm/odm with type safe projections. If not, I shall go to the typing mailing list and ask for help there.

[euresti]

I did start working on a PR to move the plugin over but ran into some issues. https://github.com/python-attrs/attrs/pull/744

Though I don't remember why I just closed the PR. I must've been in a fugue state.

[Tinche]

Though I don't remember why I just closed the PR. I must've been in a fugue state.

As are we all from time to time

[Tinche]

Hello again.

I've been playing around with learning Mypy internals over the weekend to see if this can be done.

First of all, I don't really understand how the Mypy attrs plugin fits with the Mypy plugin interface described at https://mypy.readthedocs.io/en/latest/extending_mypy.html#extending-mypy-using-plugins, since it doesn't really have an entry point as described there. Maybe @euresti can provide some guidance?

In any case, following the plugin docs and reading the Mypy source code, I created a plugin to return better data from attr.fields. Here's the first version: https://gist.github.com/Tinche/29afe1971fe8d6e5fda3ee4ca2ebd477

Here's what it does:

from attr import define, fields as f

@attr.define
class C:
    a: int
    b: float
    c: str
    d: int


reveal_type(attr.fields(C).c)  # note: Revealed type is "attr.Attribute[builtins.str]"

Neat! Here's what we can do with it:

from typing import TypeVar, Type

T = TypeVar("T")
A1 = TypeVar("A1")
A2 = TypeVar("A2")


def select_projection(
    model: Type[T], *, projection: tuple[Attribute[A1], Attribute[A2]]
) -> tuple[A1, A2]:
    pass  # Imagine an implementation here

res = select_projection(C, projection=(f(C).a, f(C).b))
reveal_type(res)  # note: Revealed type is "Tuple[builtins.int*, builtins.float*]"

Et voilà! The foundations for the first Python ORM/ODM with type-safe projections!

A few questions. I know the plugin as submitted is very rough, since I don't really know Mypy that well. Since it doesn't use the same interface as the existing plugin, how do we integrate them? Also, I'm thinking it would be useful to parametrize attr.Attribute further - right now it only has one generic type, the type of the field. It would also be useful to parametrize it by the class it belongs to, and possibly a literal for it's actual name in the class.

So the type of f(C).a instead of being attr.Attribute[int] would be attr.Attribute[C, int, Literal['a']]. I think this only requires changing the stubs and modifying the plugin. This extra data would be useful down the line in more sophisticated Mypy plugins. Imagine the select_projection function returning a TypedDict instead of a tuple. An ORM could write a Mypy plugin, relatively easily, to let Mypy know the return type of that function would be a TypedDict with two fields, typed as int and float.

[Tinche]

Having figured out how the Mypy attrs plugin works a little, I've done some work on this: https://github.com/python/mypy/pull/10467

[euresti]

Sorry for not chiming in earlier. Because the attrs plugin is part of mypy it doesn't quite follow the instructions in "Extending mypy using plugins". But it looks like you figured it out. I'll take a look at your PR over in mypy tomorrow.

[Tinche]

Thanks David, no worries - we all have lives. I'd be eager to receive feedback whenever.