cpython
cpython copied to clipboard
python
The iteration behavior of IntFlag has changed backwards-incompatibly in python 3.11 and the published documentation is now incorrect
At the Python documentation for Enum at https://docs.python.org/3/library/enum.html, the first paragraph contains the definition of an enum:
An enumeration:
- is a set of symbolic names (members) bound to unique values
- can be iterated over to return its members in definition order
- uses call syntax to return members by value
- uses index syntax to return members by name
The second bullet is no longer the case in Python 3.11. The demonstration below passes on Python 3.10 and earlier, fails on Python 3.11:
from enum import Enum, IntFlag
class MyEnum(Enum):
VAL1 = 1
VAL2 = 2
VAL3 = 3
class MyIntFlag(IntFlag):
VAL1 = 1
VAL2 = 2
VAL3 = 3
assert list(MyEnum) == [MyEnum.VAL1, MyEnum.VAL2, MyEnum.VAL3]
assert list(MyIntFlag) == [MyIntFlag.VAL1, MyIntFlag.VAL2, MyIntFlag.VAL3], list(MyIntFlag)
on Python 3.10, the script succeeds without output.
output on Python 3.11:
$ /opt/python-3.11.0/bin/python3 test3.py
Traceback (most recent call last):
File "/home/classic/dev/sqlalchemy/test3.py", line 17, in <module>
assert list(MyIntFlag) == [MyIntFlag.VAL1, MyIntFlag.VAL2, MyIntFlag.VAL3], list(MyIntFlag)
AssertionError: [<MyIntFlag.VAL1: 1>, <MyIntFlag.VAL2: 2>]
I would think this might be a regression in Python 3.11, but seeing as there are lots of changes in IntEnum overall, maybe this was intended. It does seem to be counter to the usual spirit of the standard library to change an explicitly documented behavior like this.
The change is to Flag and IntFlag: in plain enums the canonical name is the one seen first for any particular value, and an alias is a different name for that same value; in flags the canonical name is the first one seen for a power-of-two value, and aliases are different names for that same value, or names of more than one power-of-two value.
So in the example above, MyIntFlag.VAL3 is an alias for MyIntFlag.VAL1 | MyIntFlag.VAL2.
Is MyIntFlag.VAL3 a member of the enum? It's still present in __members__. If it's a member, then the phrase, "can be iterated over to return its members in definition order" would appear that it needs to be changed and this is a backwards incompatible change.
if it's not a member, then why is it in __members__. That is, I really think this change is a mistake.
it's also not in dir(), which IMO is unintuitive (also a behavioral change).
class MyIntFlag(IntFlag):
VAL1 = 1
VAL2 = 2
VAL3 = 3
VAL4 = 4
print(dir(MyIntFlag))
for python 3.10:
['VAL1', 'VAL2', 'VAL3', 'VAL4', '__class__', '__doc__', '__members__', '__module__']
python 3.11:
['VAL1', 'VAL2', 'VAL4', '__abs__', '__add__', '__and__', '__bool__', '__ceil__', '__class__', '__contains__', '__delattr__', '__dir__', '__divmod__', '__doc__', '__eq__', '__float__', '__floor__', '__floordiv__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getnewargs__', '__getstate__', '__gt__', '__hash__', '__index__', '__init__', '__init_subclass__', '__int__', '__invert__', '__iter__', '__le__', '__len__', '__lshift__', '__lt__', '__members__', '__mod__', '__module__', '__mul__', '__name__', '__ne__', '__neg__', '__new__', '__or__', '__pos__', '__pow__', '__qualname__', '__radd__', '__rand__', '__rdivmod__', '__reduce__', '__reduce_ex__', '__repr__', '__rfloordiv__', '__rlshift__', '__rmod__', '__rmul__', '__ror__', '__round__', '__rpow__', '__rrshift__', '__rshift__', '__rsub__', '__rtruediv__', '__rxor__', '__setattr__', '__sizeof__', '__str__', '__sub__', '__subclasshook__', '__truediv__', '__trunc__', '__xor__', 'as_integer_ratio', 'bit_count', 'bit_length', 'conjugate', 'denominator', 'from_bytes', 'imag', 'numerator', 'real', 'to_bytes']
This is quite a signifcant change, AFAIK there was no deprecation warning, I'm not sure where this is documented either
Compared to an enum, 3.11 now matches 3.10 and earlier:
class Kind(IntEnum):
A = 1
B = 2
C = 3
D = 1
dir(Kind)
# ['A', 'B', 'C', ...]
There is no D because it is an alias
Kind.__members__
# mappingproxy({'A': <Kind.A: 1>, 'B': <Kind.B: 2>, 'C': <Kind.C: 3>, 'D': <Kind.A: 1>})
D is there because it's a valid name, but it returns Kind.A.
The docs should be corrected, possibly by inserting 'canonical' into 'iterated over to return its members'. Flags did not have aliasing properly defined and implemented until 3.11, although enums did since they appeared in 3.4.
OK so aliases are returned in __members__, and not in dir() or iter(), is that accurate?
Also this is definitely a behavioral change either way and should be noted.
from my end it of course doesn't matter, I have to change my code regardless since we support all Python versions since 3.7.