Infer attributes from __new__

[JukkaL]

trafficstars

In this example (from #982) we define an attribute via assignment in __new__:

class C(object):
    def __new__(cls, foo=None):
        obj = object.__new__(cls)
        obj.foo = foo
        return obj

x = C(foo=12)
print(x.foo)

Currently mypy doesn't recognize attribute foo and complains about x.foo. To implement this, we could do these things:

• Infer the fact that obj is something similar to self because of the way is created via object.__new__. This should happen during semantic analysis so that this will work in unannotated method as well.
• Infer attributes from assignments via obj because it is classified as similar to self.
• Also recognize object.__new__ calls via super(...).

[JukkaL]

This is follow-up to #982.

[JukkaL]

Decreased priority since this doesn't seem like users often struggle with this -- it's easy enough to work around by inserting type annotations.

[toolforger]

One use case is NamedTuple. It's turning mypy into a noise generator for me. Not sure how this data point affects priorization.

[JelleZijlstra]

@toolforger how are you using NamedTuple? I don't think the example from the original post in this issue is applicable to NamedTuple.

[toolforger]

Using a NamedTuple subclass, subclass it with a __new__ override to provide standard parameters. Details in #5194, which is a duplicate of #1279.

[ilevkivskyi]

Using a NamedTuple subclass, subclass it with a __new__ override to provide standard parameters.

Use instead:

class Foo(NamedTuple):
    x: int
    y: int = 0

Foo(1)  # OK, same as Foo(1, 0)
Foo(1, 2)  # OK

works on Python 3.6+. If you are on Python 2, then wait for this issue to be solved.

[toolforger]

@ilevkivskyi your advice works for direct subclasses. My use case is for an indirect subclass. (I already had to move all fields up into the root class because NamedTuples don't properly support subclassing, but I cannot use frozen dataclasses until 3.7 comes out for my distro. I'm on the verge of ripping out mypy again, or switching from Python to Kotlin.)

[gvanrossum]

@toolforger I hear your frustration, but please don't take it out on us. We have a lot on our plate and we're only a small team (half of us are volunteers). This will eventually get fixed. (If you want to help by submitting a PR that might speed it up of course.)

[toolforger]

Heh. Getting advice that's inapplicable because the person didn't really understand the situation, after spending weeks of my free time on a dozen or so approaches - well what can I say, it was just the last straw.

Won't work on Python, sorry:

• I'd be just another volunteer;
• given my tribute to the ecosystem already (two years on SymPy I believe);
• deep philosophical differences about what a language should be, I'd have to bite my tongue constantly;
• already other projects on my desk anyway (I'm using Python to get something off my desk to I can work on the tool for the project I'm really interested in - I guess this rings with many people here ;-) ).

[gvanrossum]

Can we raise the priority here? I encountered several cases of this in S (links provided at request). What's worse, I also get an error on the super __new__ call:

class C(str):
    def __new__(cls) -> C:
        self: C = str.__new__(cls, 'foo')  # E: Too many arguments for "__new__" of "object"
        self.foo = 0  # E:  "C" has no attribute "foo"
        return self
C().foo  # E: "C" has no attribute "foo"

The workaround is redundant class-level attribute declarations.

[ilevkivskyi]

The super error is unrelated, it is just typeshed problem, str doesn't have __new__ there.

[gvanrossum]

The super error is unrelated, it is just typeshed problem, str doesn't have new there.

Okay, but the main problem is the need for duplication for every attribute.

[JukkaL]

Updated priority to high.

[mjpieters]

How would this work with an Enum class that uses the tuple value -> separate arguments to __new__ option?

E.g.:

from enum import Enum

class Colours(Enum):
    #        letter, ANSI base color, reverse flag
    black  = "k", "white", True
    blue   = "b", "blue"
    orange = "o", "yellow"
    red    = "r", "red"

    def __new__(cls, value: str, colour: str, reverse: bool = False):
        member = object.__new__(cls)
        member._value_ = value
        member.ansi_colour = colour
        member.reverse = reverse
        return member

The above causes several issues for MyPy; for the above example the .value type is assumed to be Tuple[str, str, bool], but if I changed the order of the definitions it could also be Tuple[str, str], and reports a Missing positional argument "colour" in call to "Colours" issue.

I can work around these issues by giving default values any arguments beyond value, and by adding conditional variable annotations:

from enum import Enum
from typing import TYPE_CHECKING

class Colours(Enum):
    #        letter, ANSI base color, reverse flag
    black  = "k", "white", True
    blue   = "b", "blue"
    orange = "o", "yellow"
    red    = "r", "red"

    if TYPE_CHECKING:
        value: str
        ansi_colour: str
        reverse: bool

    def __new__(cls, value: str, colour: str = "white", reverse: bool = False):
        member = object.__new__(cls)
        member._value_ = value
        member.ansi_colour = colour
        member.reverse = reverse
        return member

or, and that's a heavier hammer, put the __new__ definition behind an else: branch for the if TYPE_CHECKING: check. I'm much rather have mypy recognise the above as a valid usecase, however.

[tpvasconcelos]

Hiding my comment since it doesn't add anything to the discussion and I cannot remember why I posted it in the first place... (Note to future self: next time add more context and a minimal working example of what the exact issue I encountered is)

---

Is there an update on this? Or do you have a suggestion on what would be the most idiomatic way to solve the issue for now?

Cheers 🐍

[berzi]

@tpvasconcelos I worked around the issue by using setattr() instead of assigning the attribute directly:

class MyEnum(Enum):
    ONE = 1
    TWO = 2
    THREE = 3

    def __new__(cls, value, is_cool_number):
        member = object.__new__(cls)
        member._value_ = value
        # member.is_cool = is_cool_number  # Not this
        setattr(member, "is_cool", is_cool_number)  # This
 
        return member

Note that this will make Mypy happy inside __new__, but it might still complain elsewhere about is_cool not being defined on MyEnum. You still have to annotate it as a class attribute to avoid that (is_cool: bool).

And just as a reminder, remember that if your enum inherits from a different type (e.g. MyEnum(str, Enum)) you have to do the same inside __new__: member = str.__new__(cls).

It would still be nice to see this fixed upstream one day, since the issue is seven years old at this point. At the moment I don't have the time to learn mypy's codebase and contribute myself, unfortunately.

[alexei]

The above doesn't work for me (mypy 1.4.1 on Python 3.11.3):

from enum import Enum


class Num(Enum):
    WOM = (1, True)
    TOO = (2, False)
    TEE = (3, True)
    FOR = (4, False)

    def __new__(cls, value, is_cool_number):
        obj = object.__new__(cls)
        obj._value_ = value
        obj.is_not_cool = not is_cool_number
        setattr(obj, "is_cool", is_cool_number)
        return obj


Num(1)  # Missing positional argument "is_cool_number" in call to "Num"  [call-arg]
Num["TOO"]
Num.TEE.is_not_cool  # "Num" has no attribute "is_not_cool"  [attr-defined]
Num.FOR.is_cool  # "Num" has no attribute "is_cool"  [attr-defined]

The "Missing positional argument "is_cool_number" in call to "Num" [call-arg]" is a different beast, though...

Later edit: it looks like I missed an important bit in the above comment:

this will make Mypy happy inside __new__, but it might still complain elsewhere about is_cool not being defined

[brianmedigate]

if I work around this by manually annotating the member attributes, it breaks exhaustiveness checking because mypy thinks the attributes are enum elements:

from enum import Enum
from typing import assert_never


class Colours(Enum):
    #        letter, ANSI base color, reverse flag
    black  = "k", "white", True
    blue   = "b", "blue"
    orange = "o", "yellow"
    red    = "r", "red"

    ansi_colour: str
    reverse: bool

    def __new__(cls, value: str, colour: str, reverse: bool = False):
        member = object.__new__(cls)
        member._value_ = value
        member.ansi_colour = colour
        member.reverse = reverse
        return member

c: Colours  = Colours.red

if c is Colours.black or c is Colours.blue or c is Colours.orange or c is Colours.red:
    pass
else:
    assert_never(c)

results in:

error: Argument 1 to "assert_never" has incompatible type "Literal[Colours.ansi_colour, Colours.reverse]"; expected "NoReturn"  [arg-type]

am I missing something? is there a way to avoid this?

This is on python 3.11.9 and mypy 1.10.1

[garkin]

Off-topic, but closest issue i got from google to my problem. Someone could find it helpful. This works for me in both Mypy and Pyright. __new__ should return an object explicitly casted to Popik.

class Popik:
    def __init__(self, *args, **kwargs):
        # __init__() would be called after __new__()
        #   should have same arguments as __new__() 
        #   or use (*args, **kwargs) stub
        self.booty: str

    def __new__(cls):
        # in reality i hijack and extend object from extrenal factory
        obj: Popik = cast(Any, type("stub", (object,), {})()) 
        obj.__class__ = cls
        obj.booty = "mignon"
        return obj

p = Popik()
print(p.booty)