[REF-759] TypeError: Object of type list is not JSON serializable

[masenf]

Describe the bug Attempting to serialize a mutable type as JSON inside an event handler does not work.

To Reproduce

import json

import reflex as rx


class State(rx.State):
    foo: list[int] = [1, 2, 3]

    @rx.var
    def foo_json(self) -> str:
        return json.dumps(self.foo)


def index() -> rx.Component:
    return rx.text(State.foo_json)


app = rx.App()
app.add_page(index)
app.compile()

Expected behavior The page should display [1, 2, 3]. Instead there is an error on the console

Task exception was never retrieved
future: <Task finished name='Task-32' coro=<AsyncServer._handle_event_internal() done, defined at /Users/masenf/code/reflex-dev/VENV-dev/lib/python3.11/site-packages/socketio/asyncio_server.py:522> exception=TypeError('Object of type list is not JSON serializable')>
Traceback (most recent call last):
  File "/Users/masenf/code/reflex-dev/VENV-dev/lib/python3.11/site-packages/socketio/asyncio_server.py", line 524, in _handle_event_internal
    r = await server._trigger_event(data[0], namespace, sid, *data[1:])
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/masenf/code/reflex-dev/VENV-dev/lib/python3.11/site-packages/socketio/asyncio_server.py", line 569, in _trigger_event
    return await self.namespace_handlers[namespace].trigger_event(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/masenf/code/reflex-dev/VENV-dev/lib/python3.11/site-packages/socketio/asyncio_namespace.py", line 37, in trigger_event
    ret = await handler(*args)
          ^^^^^^^^^^^^^^^^^^^^
  File "/Users/masenf/code/reflex-dev/reflex/reflex/app.py", line 975, in on_event
    async for update in process(self.app, event, sid, headers, client_ip):
  File "/Users/masenf/code/reflex-dev/reflex/reflex/app.py", line 805, in process
    update = await app.preprocess(state, event)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/masenf/code/reflex-dev/reflex/reflex/app.py", line 271, in preprocess
    out = await middleware.preprocess(app=self, state=state, event=event)  # type: ignore
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/masenf/code/reflex-dev/reflex/reflex/middleware/hydrate_middleware.py", line 54, in preprocess
    delta = format.format_state({state.get_name(): state.dict()})
                                                   ^^^^^^^^^^^^
  File "/Users/masenf/code/reflex-dev/reflex/reflex/state.py", line 1047, in dict
    {
  File "/Users/masenf/code/reflex-dev/reflex/reflex/state.py", line 1049, in <dictcomp>
    prop_name: self.get_value(getattr(self, prop_name))
                              ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/masenf/code/reflex-dev/reflex/reflex/state.py", line 626, in __getattribute__
    value = super().__getattribute__(name)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/masenf/code/reflex-dev/reflex/reflex/vars.py", line 1286, in __get__
    return super().__get__(instance, owner)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/masenf/code/reflex-dev/repro-json-serialize/repro_json_serialize/repro_json_serialize.py", line 11, in foo_json
    return json.dumps(self.foo)
           ^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/encoder.py", line 200, in encode
    chunks = self.iterencode(o, _one_shot=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/encoder.py", line 258, in iterencode
    return _iterencode(o, 0)
           ^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/encoder.py", line 180, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type list is not JSON serializable

Specifics (please complete the following information):

• Python Version: 3.11.4
• Reflex Version: main
• OS: macOS 13.5.2

Additional context Changing the code to return json.dumps(self.foo.__wrapped__) works.

_REF-759

[shashank40]

Can you please assign it to me? @masenf ?

[panxiaoguang]

I found the same issues, but for a backend-only variable, a backend-only variable is still a reflex state. MutableProxy instead of a Python variable. It's quite strange because it will raise errors when I try to put it into a function that should be JSON serialized.

[masenf]

The workaround is to access the value through self.get_value to remove the MutableProxy.

The MutableProxy is there to track changes to mutable types and determine which vars need to be updated. Even backend vars are wrapped in MutableProxy, because computed vars may depend on them.

If your var is called _my_value: list[str], then access it like this when serializing:

def handler(self):
    some_api_call(data=self.get_value(self._my_value))

We are looking for a general way to resolve this issue that doesn't involve hacking the json encoder, please let us know if you have any ideas.

[panxiaoguang]

The workaround is to access the value through self.get_value to remove the MutableProxy.

The MutableProxy is there to track changes to mutable types and determine which vars need to be updated. Even backend vars are wrapped in MutableProxy, because computed vars may depend on them.

If your var is called _my_value: list[str], then access it like this when serializing:

def handler(self):
    some_api_call(data=self.get_value(self._my_value))

We are looking for a general way to resolve this issue that doesn't involve hacking the json encoder, please let us know if you have any ideas.

Thank you so much, that's working fine!