[Python] FBS Enums are of underlying type, not of enum type

[sethraymond]

Steps to Reproduce

1. Create the following flatbuffers files: an_enum.fbs:

enum AnEnum : int {
    FOO = 0,
    BAR = 1
}

test.fbs

include "an_enum.fbs";

table Test {
    bar: AnEnum;
}

root_type Test;

2. Generate Python code using flatc --python --gen-object-api --gen-compare test.fbs an_enum.fbs
3. Observe the generated code

Expected Behavior In the object API, the Test.bar property is of type AnEnum and the AnEnum is an enum object with underlying type int, i.e. AnEnum.py

from enum import IntEnum

class AnEnum(IntEnum):
    FOO = 0,
    BAR = 1

Test.py

import AnEnum
# ...
class TestT(object):

    # TestT
    def __init__(self):
        self.bar = AnEnum.FOO  # type: AnEnum.AnEnum

Actual Behavior Test.py doesn't even import AnEnum.py and the Test.bar type is simply of type int, meaning that any value can be put into the object.

[sethraymond]

This could maybe be a good use case for the --scoped-enums flag available for C++.

[fliiiix]

It does exist already but its considered typing info so you need to add the flags --python-typing and --python-version 3.12 (the important bit here is the 3 in the beginning because IntEnum did not exist in Python 2 so you could also do --python-version 3)

And then you get extra files with type info in the form of .pyi files.

$ cat AnEnum.pyi
from __future__ import annotations

import flatbuffers
import numpy as np

import flatbuffers
import typing
from enum import IntEnum

uoffset: typing.TypeAlias = flatbuffers.number_types.UOffsetTFlags.py_type

class AnEnum(IntEnum):
  FOO: int
  BAR: int

[jtdavis777]

I've been doing a bit of digging on this and I think it may be valuable to also support generating first class enum types over object types -- see #5083

My two cents are that we could update the generated enum class to be a true enum, but leave the setters and getters alone to handle raw ints. This would preserve compatibility but give additional functionality above the type hints, and users can choose to convert values into these enum types (and handle errors) if they so choose.

It doesn't seem like only modifying the enum's class type and type info (and leaving all other generated code/type info alone) would be a breaking change, but I'm not a python or flatbuffers-python-user expert.

This should definitely be gated behind --python-version 3 and (probably?) should be gated behind a new flag (--python-enum maybe?)

@fliiiix would love your input here, and I'm happy to make a PR if you agree.

Linking the original PR for reference: #8145

[fliiiix]

I think it may be valuable to also support generating first class enum types

What would be the use-cases enabled by this? The only thing i see is the Enum name to string eg. Enum.A.name -> 'A' but i personally don't even see where this would be useful.

but leave the setters and getters alone to handle raw ints

I think one needs to be careful to get this right because you probably still want to support things like comparing with ints eg Enum.A == 1 should still be true

It doesn't seem like only modifying the enum's class type and type info would be a breaking change

I am not so sure about that part i think it has at least the potential to introduce interesting new behavior 🤷. I guess we would need to implement this behind a flag. Im also not sure how many flags we want to add or what the strategy is here. Right now a lot of flags are required to get something somewhat modern. But that might be a different issue.

[jtdavis777]

What would be the use-cases enabled by this? The only thing i see is the Enum name to string eg. Enum.A.name -> 'A' but i personally don't even see where this would be useful.

this is exactly the use case I'm thinking of, and I have a need for it -- I send flatbuffers over ethernet between some applications, and have built a python monitoring/maintenance tool where I want it to present the human readable names in the console and log.

I think one needs to be careful to get this right because you probably still want to support things like comparing with ints eg Enum.A == 1 should still be true

IntEnum inherits from int, and therefore these comparisons still work. I also think I would want to change the python setters to accept these enum types instead of raw ints, but would want to leave the getters alone. this would provide some type and range safety to the serialization API.

Agree that python flags are getting a bit out of hand, I would prefer to just have this gated by python major version == 3, which I think would be fine if it doesn't break any APIs. but maybe can look into solving that problem in a separate issue and just create a flag here.

[jtdavis777]

I should rephrase, the use case I need is to be able to do something like

def print_value(val: int) -> None:
    print(f"{MyEnum(val).name}")

and also the opposite can be handy:

def get_value(name: str) -> MyEnum:
    return MyEnum[name]

[jtdavis777]

Funny note - I instrumented this in my code base to play with, and the only thing it broke were my own custom to and from string functions. Haha