`Content` object not JSON serializable

[douglaslwatts]

Description of the bug:

When attempting to use the Content object in API request/response there is an error that the object is not JSON serializable.

I have created a small example API so the issue can easily be reproduced. The README has instructions for running the API and how to reproduce the issue.

github.com/douglaslwatts/gemini_bug_example_api

Actual vs expected behavior:

Expected Behavior: The list[Content] history would be successfully serialized for the request/response bodies

Actual Behavior: We get the below error

[2024-03-22 09:04:03,805] ERROR in app: Exception on /api/v1/chat-message [POST]
Traceback (most recent call last):
  File ".pyenv/versions/3.12.2/lib/python3.12/site-packages/flask/app.py", line 1463, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".pyenv/versions/3.12.2/lib/python3.12/site-packages/flask/app.py", line 873, in full_dispatch_request
    return self.finalize_request(rv)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".pyenv/versions/3.12.2/lib/python3.12/site-packages/flask/app.py", line 892, in finalize_request
    response = self.make_response(rv)
               ^^^^^^^^^^^^^^^^^^^^^^
  File ".pyenv/versions/3.12.2/lib/python3.12/site-packages/flask/app.py", line 1183, in make_response
    rv = self.json.response(rv)
         ^^^^^^^^^^^^^^^^^^^^^^
  File ".pyenv/versions/3.12.2/lib/python3.12/site-packages/flask/json/provider.py", line 214, in response
    f"{self.dumps(obj, **dump_args)}
", mimetype=self.mimetype
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".pyenv/versions/3.12.2/lib/python3.12/site-packages/flask/json/provider.py", line 179, in dumps
    return json.dumps(obj, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".pyenv/versions/3.12.2/lib/python3.12/json__init__.py", line 238, in dumps
    **kw).encode(obj)
          ^^^^^^^^^^^
  File ".pyenv/versions/3.12.2/lib/python3.12/json/encoder.py", line 200, in encode
    chunks = self.iterencode(o, _one_shot=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".pyenv/versions/3.12.2/lib/python3.12/json/encoder.py", line 258, in iterencode
    return _iterencode(o, 0)
           ^^^^^^^^^^^^^^^^^
  File ".pyenv/versions/3.12.2/lib/python3.12/site-packages/flask/json/provider.py", line 121, in _default
    raise TypeError(f"Object of type {type(o).__name__} is not JSON serializable")
TypeError: Object of type Content is not JSON serializable
127.0.0.1 - - [22/Mar/2024 09:04:03] "POST /api/v1/chat-message HTTP/1.1" 500 -

Any other information you'd like to share?

When using the chat-bison models, we can return the chat session history in the response body, and accept it in the request body with success using a list of ChatMessage, which is the history data structure used for those models. However, when we try this using Gemini, there is an error stating that the Content object, which is the history data structure for Gemini models, is not JSON serializable.

The intention is that the caller is another application and the chat history is stored/managed there. If any further conversation with the chat history is needed, then that history can be passed in subsequent requests. This is working in a real world application using the chat-bison 32K model but the plan is to change over to using the Gemini Pro model.

[douglaslwatts]

I am just checking in on this issue. Has there been any progress here?

[getwithashish]

I had to store the Content object in my database. A quick hack I did was to use jsonpickle to serialize the object into JSON.

[douglaslwatts]

I have found that this custom field serializer on the response model is a workaround that allows returning a usable response.

    @field_serializer('history')
    def serialize_history(self, history: list[Content], _info):
        return [
            {
                'role': hist.role,
                'parts': [
                    {
                        'text': part.text
                    }
                    for part in hist.parts
                ]
            }
            for hist in history
        ]

However, it does not work for the request model. I have not yet come up with a custom field serializer that is a workaround for that. It would still be best if the Content and Part objects were themselves JSON serializable, so a workaround would not be needed, and Pydantic could handle them as it does the ChatMessage object that is used for the history with the Chat Bison models.

[cspollar]

Depending on your goals, this custom serializer may be a little cleaner.

    @field_serializer('history')
    def serialize_history(self, history: list[Content]):
        return [x.to_dict() for x in history]