Add function to "instantiate" __annotations__-derived 'attr.ib's.

[asford]

TLDR

It would be very useful to extract the __annotations__ handling logic currently in _transform_attrs into a standalone function that can be used to convert annotations into _CountingAttrs and _CountAttr.types. This would support attrs-based plugins that set or modify attribute validators/converters/metadata before attr.s-based class generation.

Current State

One major strength of attrs is the ability to layer metadata, validation and conversion logic onto attr classes to implement project-specific class declaration patterns.

As a toy example, consider the case of adding simple init-time type validation to a class:

@attr.s
class Foo:
    bar = attr.ib(type=str, validator=attr.validators.instance_of(str))
    bat = attr.ib(type=int, validator=attr.validators.instance_of(int))
    baz = attr.ib(type=float, validator=attr.validators.instance_of(float))

Very punchy, but we're spoiled. This can be dry-ed out with a trivial decorator:

def validate_attr_types(cls):
    counting_attrs = [
	v for v in cls.__dict__.values()
	if isinstance(v, attr._make._CountingAttr)
    ]
    
    for ca in counting_attrs:
	if ca.type:
	    ca.validator(attr.validator.instance_of(ca.type))
	    
    return cls
    
@attr.s
@validate_attr_types
class Foo:
    bar = attr.ib(type=str)
    bat = attr.ib(type=int)
    baz = attr.ib(type=float)

As our decorator is in the business of modifying the declared attributes to add validation (or conversion, metadata, ...) it needs to run before attr.s, which freezes _CountingAttrs down into Attributes. This isn't an obstacle, as we're declaring all the attr.ibs and most of the time we'll define a nice wrapper around attr.ib if we need any non-trivial logic.

The Problem

It's 2018 and type annotations are in! We would really prefer to write something like:

@attr.s(auto_attribs=True)
@validate_attr_types
class Foo:
    bar : str
    bat : int
    baz : float 

Unfortunately this gets us into a real sticky wicket. At the time our decorator fires Foo doesn't have any _CountingAttr instances available for us to update, just entries in the __annotations__ dict. The naive idea of swapping the decorator ordering is no good, as the generated attributes would be frozen by attr.s. We might be able to let attr.s run, hackily create an updated variant of the class, call attr.s again, and then return the second generated class but this just feels wrong.

Slightly subtly, this problem bites us even if we mandate that all attr.ibs are declared:

@attr.s(auto_attribs=True)
@validate_attr_types
class Foo:
    bar : str = attr.ib()
    bat : int = attr.ib()
    baz : float = attr.ib()

Our decorator will now fail because the type information stored in the __annotations__ hasn't yet been extracted into _CountingAttr.type.

A Solution

It would be ideal if attrs offered an idempotent function to convert all __annotations__ based attribute information into concrete, mutable _CountingAttr instances on a pre-attr.s class. This would convert:

class FooPre:
    bar : str
    bat : int = attr.ib()
    baz = attr.ib(type=float)

into _CountingAttr equivalents of:

class Foo:
    bar = attr.ib(type=str)
    bat = attr.ib(type=int)
    baz = attr.ib(type=float)

Given the straw-man name attr.transform_annotations, this would let us write our simple decorator as:

def validate_attr_types(cls):
    
    # in-place update of cls from __annotations__
    attr.transform_annotations(cls, auto_attribs=True)

    counting_attrs = [
	v for v in cls.__dict__.values()
	if isinstance(v, attr._make._CountingAttr)
    ]
    
    for ca in counting_attrs:
	if ca.type:
	    ca.validator(attr.validator.instance_of(ca.type))
	    
    return cls

and write:

@attr.s
@validate_attr_types
class Foo:
    bar : str
    bat : int
    baz : float

[wsanchez]

Note that with type annotations, instance_of is a partial solution, since we'll also see people trying interface types:

@attr.s
@validate_attr_types
class Foo:
    bar : AnyStr
    bat : Optional[float]
    baz : Mapping[str, int]

While instance_of validation is useful, I suspect, we'll see lots of bug reports right away.

Maybe if validate_attr_types gets a more limited-sounding name we might see fewer.

[wsanchez]

(Excellent write-up, by the way.)

[asford]

Just to clarify, the example decorator is just a minimal immediately-grok-able sample. I'm not proposing that such a decorator be included in attrs, and have (hopefully) clarified this in the example.

The actually-motivating use case is a little more complex; I'm attempting to cleanup and spinout an attrs-based tensor data adapter we're using in a private project. This component currently relies on a hack-y combination of a mixed-in base class and method-time analysis of the attribute list, but I would like to convert it to a simpler & more performant model based on an an attr.s style class decorator. This will require definition-time processing of the target class's attributes to set library-specific metadata values.

[Kentzo]

Perhaps a solution that's more inline with decorator-esque customization is to provide decorators that would split attrs into transforming and freezing:

@attr.freeze
@validate_attr_types
@attr.transform
class Foo:
    bar : str
    bat : int
    baz : float

[hynek]

Just to TL;DR it, you would like to change our flow to make

class Foo:
    bar = attr.ib(type=str)

the canonical form that is used by attrs itself and make type annotations a separate, idempotent step?

I guess that might even simplify our internal code to a certain degree? 🤔

[asford]

I would slightly expand that to state that there should be a canonical "internal" form, which both attrs and attrs-based external libraries can rely on during class configuration. As this canonical form won't match the preferred user-facing form, which is likely a "dataclass" style declaration using type annotations, attrs should provide an idempotent coercion function to convert from all allowable using-facing formats to the canonical programmatic representation.

To lift out specifics, this function would be an subset of the current attr._make._transform_attrs function however it would involve an in-place modification of the target class to add _CountingAttr class properties rather than extracting an attribute list.

This may slightly simplify the internal code, but I don't believe it will have much of an effect. The _transform_attrs call is the first step of _ClassBuilder process, so attrs is effectively relying on a standard representation throughout class construction already.

[asford]

@Kentzo I don't think this a bad idea, but it seems like it'd unnecessarily expose an implementation detail about attrs' preferred internal representation to the consuming user? If we write an idempotent transformation function then attrs, and attrs-dependent wrapping libraries, can call that function before any class modification to ensure that the canonical internal representation is present.

[asford]

As a gut-check, does this idea seem generally acceptable? If so I can open a straw-man PR to help focus the conversation around implementation specifics.

[sscherfke]

This looks related to #649 and #653

[euresti]

Don't forget that annotations could be strings too. And in Python 3.10 will always be strings. And so not only do you have to evaluate them, you might also have forward references to objects that don't exist when the class is being created. Here's what we do at our company. (Someday we will open source this):

We have a decorator that replaces attr.s and before it calls attrs.s shoves an

def __attrs_post_init__(self: object) -> None:
            _verify_attribute_types(self)

into the class.

_verify_attribute_types first calls something like: https://github.com/python-attrs/attrs/pull/302/files#diff-21756223332b04173cec8910fda6655bR979 (And caches it on the class for performance)

And now that it has the real types it loops through the arguments and checks them using whatever method you want.

Edit: correct link

[hynek]

FYI the linked doc doesn't seem to be public.

[euresti]

Oops. Not what I meant to post.

[euresti]

Like this?

import attr
from attr import NOTHING, attrib
from typing import TypeVar

from attr._make import _get_annotations, _CountingAttr, _is_class_var

_C = TypeVar("_C", bound=type)

def make_auto_attribs(cls: _C) -> _C:
    for attr_name, type in _get_annotations(cls).items():
        if _is_class_var(type):
            continue
        a = getattr(cls, attr_name, NOTHING)
        if not isinstance(a, _CountingAttr):
            setattr(cls, attr_name, attrib(default=a))
    return cls


@make_auto_attribs
class Foo:
    x: int
    y: str = "15"
    z: bool = attr.ib(True)

print(Foo.x)
print(Foo.y)
print(Foo.z)

[hynek]

I guessed I asked this before, but is there a way we could add this to attrs behind an option? It seems very handy especially when looking towards 3.10.

[euresti]

Which part is "this" exactly? There are lots of proposals in this issue. :)

[hynek]

Heh sorry. 😅 I meant automatically running something like make_auto_attribs such that the typing information is always concrete.

[asford]

If I understand this correctly, do you mean something like make_auto_attribs accompanied by something like resolve_types from #302 so that the type property is resolved into a concrete value?

My interpretation of make_auto_attributes as implemented above is that it tackles the top-level request, but that it does not handle resolving typing information from strings to concrete types.

[euresti]

resolving types often cannot be performed at class creation time. (e.g. if you have forward refs) Also not everything needs these to be concrete types so it would be wasted resources to do it automatically.

[hynek]

I'm super sorry, I'm sure you told me that (at least) once before. Could I compel you to submit a PR to explain this problematic in https://www.attrs.org/en/stable/types.html because we're 100% gonna get bugs about it and also I'll be able to refer to your explanation the next time I forget. 🙈😅