mashumaro icon indicating copy to clipboard operation
mashumaro copied to clipboard

Investigate support for recursive Union types

Open JWCS opened this issue 3 months ago • 2 comments

  • mashumaro version: 3.12
  • Python version: 3.12.2
  • Operating System: alpine 3.19 docker

Description

In the past, I've used the below JSON type definitions, which have worked well with both static and runtime type checkers (mypy (except the new one, wip) and beartype). I had a message type that included an arbitrary json blob as a field. Attempting to serialize that with mashumaro led to an error. I'm not entirely sure if this recursive functionality is supported by mashumaro, due to pre-compilation, only that the underlying types themselves are. If there's a better workaround, I apologies if I missed the documentation.

What I Did

I tried to use both the "new" 3.12 type keyword definition of JSON, and the "old" recursive style (avoiding from __future__ import annotations).

from mashumaro.mixins.orjson import DataClassORJSONMixin
from dataclasses import dataclass
from typing import Union
type JSON = Union[dict[str, JSON], list[JSON], str, int, float, bool, None]
@dataclass
class MsgT(DataClassORJSONMixin):
    name: str
    meta: str
    msg: JSON
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/usr/local/lib/python3.12/site-packages/mashumaro/mixins/dict.py", line 24, in __init_subclass__
    compile_mixin_unpacker(cls, **builder_params["unpacker"])
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/mixin.py", line 49, in compile_mixin_unpacker
    builder.add_unpack_method()
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/code/builder.py", line 557, in add_unpack_method
    self._add_unpack_method_lines(method_name)
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/code/builder.py", line 462, in _add_unpack_method_lines
    ).build(
      ^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/code/builder.py", line 1276, in build
    unpacked_value = UnpackerRegistry.get(
                     ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/common.py", line 238, in get
    expr = packer(spec)
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/unpack.py", line 806, in unpack_special_typing_primitive
    raise UnserializableDataError(
mashumaro.exceptions.UnserializableDataError: JSON as a field type is not supported by mashumaro
from mashumaro.mixins.orjson import DataClassORJSONMixin
from dataclasses import dataclass
from typing import Union
JSON = Union[dict[str, 'JSON'], list['JSON'], str, int, float, bool, None]
@dataclass
class MsgT(DataClassORJSONMixin):
    name: str
    meta: str
    msg: JSON
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/usr/local/lib/python3.12/site-packages/mashumaro/mixins/dict.py", line 24, in __init_subclass__
    compile_mixin_unpacker(cls, **builder_params["unpacker"])
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/mixin.py", line 49, in compile_mixin_unpacker
    builder.add_unpack_method()
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/code/builder.py", line 557, in add_unpack_method
    self._add_unpack_method_lines(method_name)
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/code/builder.py", line 462, in _add_unpack_method_lines
    ).build(
      ^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/code/builder.py", line 1276, in build
    unpacked_value = UnpackerRegistry.get(
                     ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/common.py", line 238, in get
    expr = packer(spec)
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/unpack.py", line 720, in unpack_special_typing_primitive
    return UnionUnpackerBuilder(union_args).build(spec)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/common.py", line 212, in build
    self._add_body(spec, lines)
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/unpack.py", line 169, in _add_body
    for unpacker in (
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/unpack.py", line 170, in <genexpr>
    UnpackerRegistry.get(spec.copy(type=type_arg, expression="value"))
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/common.py", line 238, in get
    expr = packer(spec)
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/unpack.py", line 1219, in unpack_collection
    f'{{{inner_expr(0, "key")}: {inner_expr(1)} '
                                 ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/unpack.py", line 1153, in inner_expr
    return UnpackerRegistry.get(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/common.py", line 238, in get
    expr = packer(spec)
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/unpack.py", line 720, in unpack_special_typing_primitive
    return UnionUnpackerBuilder(union_args).build(spec)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/common.py", line 212, in build
    self._add_body(spec, lines)
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/unpack.py", line 169, in _add_body
    for unpacker in (
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/unpack.py", line 170, in <genexpr>
    UnpackerRegistry.get(spec.copy(type=type_arg, expression="value"))
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/common.py", line 238, in get
    expr = packer(spec)
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/unpack.py", line 1219, in unpack_collection
    f'{{{inner_expr(0, "key")}: {inner_expr(1)} '
                                 ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/unpack.py", line 1153, in inner_expr
    return UnpackerRegistry.get(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/common.py", line 238, in get
    expr = packer(spec)
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/unpack.py", line 801, in unpack_special_typing_primitive
    evaluated = spec.builder.evaluate_forward_ref(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/code/builder.py", line 339, in evaluate_forward_ref
    return evaluate_forward_ref(typ, globalns, self.__dict__)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/helpers.py", line 769, in evaluate_forward_ref
    return typ._evaluate(
           ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/typing.py", line 907, in _evaluate
    eval(self.__forward_code__, globalns, localns),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 1, in <module>
NameError: name 'JSON' is not defined

JWCS avatar Apr 01 '24 19:04 JWCS