Unable to handle a self-referencing union of primitive types

[charles-dyfis-net]

Description

I'm trying to use msgspec in a (historically pydantic-based) codebase that uses a self-referencing type declaration to describe contents that can be successfully serialized as JSON. Unfortunately, msgspec fails when handling type definitions that include any values leveraging that type; a minimal reproducer follows:

import msgspec

type JsonTypes = None | str | float | bool | int | list["JsonTypes"] | dict[str, "JsonTypes"]

msgspec.json.decode(b'null', type=JsonTypes)

...which yields:

TypeError: Type ''JsonTypes'' is not supported

The problem does not take place if we modify the definition to type JsonTypes = None | str | float | bool | int | list[Any] | dict[str, Any] -- but that would defeat the point.

[jcrist]

Hmmm, I thought we raised a better error message in that case, but maybe not.

Recursive basic types aren't currently expressable in msgspec's internal type system, and refactoring to support them would be a major bit of work. When I added support for python 3.12's type annotations I punted on recursive type support because of this.

For this specific case, the output of msgspec.json.decode will always be one of those types for Any typed fields. If you're trying to constrain values in decode, then subbing in Any for JsonTypes will have the same effect and will work today.

If your goal is to get systems like pyright/mypy to know the JsonTypes, you could do this with a TYPE_CHECKING block:

from typing import TYPE_CHECKING, Any

if TYPE_CHECKING:
    type JsonTypes = None | str | float | bool | int | list["JsonTypes"] | dict[str, "JsonTypes"]
else:
    JsonTypes = Any

This should let static analysis tools still do their thing, while still working with msgspec.

[charles-dyfis-net]

Thank you! Unfortunately, pydantic is as much a target as static analysis tools are; so as necessary as the workaround may be, it seems a bit unfortunate.

Some context that my contrieved example dropped is that I don't actually require JSON (de)serialization in the use case that prompted this ticket: The issue at hand can be observed when msgspec._core.convert is called by litestar's parse_values_from_connection_kwargs. In that context, objects that can't be JSON-serialized at all (Requests &c) are passed through correctly, but the recursive type definition causes the same TypeError seen above.

A less-contrieved reproducer, then, might look like:

from litestar import Litestar, get
import litestar.di as di

type JsonTypes = None | str | float | bool | int | list["JsonTypes"] | dict[str, "JsonTypes"]

async def get_di_argument() -> JsonTypes:
    return None

@get("/")
async def async_hello_world(di_argument: JsonTypes) -> JsonTypes:
    return di_argument

app = Litestar(route_handlers=[async_hello_world], dependencies={"di_argument": di.Provide(get_di_argument)})

...triggering the stack trace, when run:

ERROR - 2024-08-21 13:38:14,162 - litestar - config - Uncaught exception (connection_type=http, path=/):
Traceback (most recent call last):
  File "/nix/store/psiw66jg5l79wacwlj6xwzb3sr2lql96-python3-3.12.4-env/lib/python3.12/site-packages/litestar/middleware/_internal/exceptions/middleware.py", line 159, in __call__
    await self.app(scope, receive, capture_response_started)
  File "/nix/store/psiw66jg5l79wacwlj6xwzb3sr2lql96-python3-3.12.4-env/lib/python3.12/site-packages/litestar/_asgi/asgi_router.py", line 99, in __call__
    await asgi_app(scope, receive, send)
  File "/nix/store/psiw66jg5l79wacwlj6xwzb3sr2lql96-python3-3.12.4-env/lib/python3.12/site-packages/litestar/routes/http.py", line 80, in handle
    response = await self._get_response_for_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/psiw66jg5l79wacwlj6xwzb3sr2lql96-python3-3.12.4-env/lib/python3.12/site-packages/litestar/routes/http.py", line 132, in _get_response_for_request
    return await self._call_handler_function(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/psiw66jg5l79wacwlj6xwzb3sr2lql96-python3-3.12.4-env/lib/python3.12/site-packages/litestar/routes/http.py", line 152, in _call_handler_function
    response_data, cleanup_group = await self._get_response_data(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/psiw66jg5l79wacwlj6xwzb3sr2lql96-python3-3.12.4-env/lib/python3.12/site-packages/litestar/routes/http.py", line 191, in _get_response_data
    parsed_kwargs = route_handler.signature_model.parse_values_from_connection_kwargs(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/psiw66jg5l79wacwlj6xwzb3sr2lql96-python3-3.12.4-env/lib/python3.12/site-packages/litestar/_signature/model.py", line 203, in parse_values_from_connection_kwargs
    return convert(kwargs, cls, strict=False, dec_hook=deserializer, str_keys=True).to_dict()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: Type ''JsonTypes'' is not supported
> /nix/store/psiw66jg5l79wacwlj6xwzb3sr2lql96-python3-3.12.4-env/lib/python3.12/site-packages/litestar/_signature/model.py(203)parse_values_from_connection_kwargs()
-> return convert(kwargs, cls, strict=False, dec_hook=deserializer, str_keys=True).to_dict()

Of course, this may be something that's more in Litestar's world than msgspec's, and I'm glad to move back over to their support channels if it's appropriate to do so -- but a means to treat objects whose types can't be understood as completely opaque would, in this context, be an entirely satisfactory fix.