Should we integrate with `annotated-types`?

[jcrist]

Follow up from #176 and #154.

We currently support adding constraints to types by annotating them with msgspec.Meta. annotated-types provides it's own annotations describing the same constraints - we could potentially integrate with them in 2 ways:

• Consume annotated-types annotations. This would let msgspec work with annotated types annotations as well as our own built-in Meta annotations.
• Produce annotated-types annotations. This would make Meta.__iter__ produce the equivalent annotated-types annotations, letting it work with other tools that consume such types.

I'm not convinced there's a real use case for either of these integrations, and would want to hear from multiple real users before expending any effort supporting this. See https://github.com/jcrist/msgspec/pull/176#issuecomment-1229265714 for a longer write up of my thoughts.

[adriangb]

So I'm obviously somewhat biased here but I do want to share some thoughts on the pros:

• As a user you get to use tools like Hypothesis out of the box with no special integration needed by Hypothesis. All this would require is that Meta.__iter__ be implemented. I think this should be pretty straightforward to do and it can even be done lazily (e.g. importing the annotated-types types within __iter__ so that it is zero-cost if this functionality is not used).
• If you're a Pydantic user, you'd be able to migrate your entire codebase with a simple regex (replace pydantic.BaseModel by msgspec.Struct, for the most part). I think given the recent emphasis on Pydantic feature parity this would be very interesting. This would require that msgspec be able to understand annotated-type which might require moving some parsing of the types from C to Python (or a lot of work in C, which you seem to handle no problem 😄 ).

[rjdbcm]

In my personal projects I have been using a fork with iter implemented (via crude hack but seems to work alright)

I have seen pyparsing used to parse a constraint string minilang into actual annotated types as in AndreaCensi/PyContracts. I would imagine implementing a stack-based language is beyond the scope of msgspec but I wanted to mention it because I had fiddled with the idea of the minilang as an actual C extension but found it a bit reinventing the wheel-y

[akotlar]

I am happy with the current system, if it works with static typing. If it doesn't, I would love for it to either be updated to do so, or if the performance hit weren't large, to support annotated-types. Here is my test case, I may be doing something wrong (constraint documentation specifies that the current system is supported by mypy):

from typing import Annotated

from msgspec import Struct, Meta

UnitFloatValidator = Annotated[
    float, Meta(ge=0.0, le=1.0)
]

Length = Annotated[list[int], Meta(max_length=3)]


class ProbabilityInterval(Struct):
    """Represent an interval of probabilities."""

    lower_bound: UnitFloatValidator
    upper_bound: UnitFloatValidator

class Length3String(Struct):
    """ A string of length 3"""

    some_string: Length

if __name__ == "__main__":

    # These don't fail, but should
    print(ProbabilityInterval(lower_bound=-1, upper_bound=1.1))
    print(Length3String(some_string=[1,2,3,4]))

    # Arguably this is ok not to fail, because Python ints and floats are compatible
    print(ProbabilityInterval(lower_bound=int(1), upper_bound=1.0))

    # These fail
    print(Length3String(some_string=["a", "b", "c"]))
    print(ProbabilityInterval(lower_bound=[1], upper_bound=1.1))

The cases that don't fail are missed by both mypy and pyright, in my hands at least.

Grateful for all the work you've done on this btw, and your willingness to engage the community.

[jcrist]

To my knowledge no static analysis tool (mypy, pyright, or otherwise) currently has support for doing anything with annotations added via an Annotated[...] (annotated-types or otherwise). One of the arguments in the PEP proposing typing.Annotated was that type checkers could ignore these extra annotations and have them still be available at runtime.

To support these kind of constructs in mypy you'd need to write a custom mypy plugin. AFAIK there's no way to add a similar plugin to pyright.

[akotlar]

Ah I misunderstood annotated-types.

I’m going to give this solution a try: https://stackoverflow.com/questions/32787411/how-to-declare-python-constraint-on-generic-type-to-support-lt

[adriangb]

Hi @jcrist a followup discussion here: https://peps.python.org/pep-0746/ and https://discuss.python.org/t/pep-746-typedmetadata-for-type-checking-of-pep-593-annotated/53834.

Even if msgspec doesn't use annotated-types I'd still really appreciate your input on that PEP since msgspec can adopt it even without annotated-types. In particular on the signature of the method (type vs. value). Thanks!

[tlambert03]

came looking to see whether there had been discussion of using annotated-types here, and found this. Just wanted to put in my support for msgspec supporting annotated-types annotations. Wouldn't mean depending on it, just basically "honoring" it such that Annotated[int, annotated_types.Ge(0)] had the same semantics as Annotated[int, msgspec.Meta(ge=0)], etc...

I'm someone who definitely likes to go back and forth between msgspec and pydantic, and also generally likes to support concepts that are re-usable and shareable. annotated-types is a nice initiative for merely marking the concept of a constraint, in a simple-to-depend-on package, while imposing almost no assumptions about implementation details. 👍

PEP 746 certainly looks nice as well :)