marimo-team
Error when using custom AI provider
Describe the bug
I am using NVIDIA Brev for hosting an LLM and want to connect my marimo instance to this. According to Brev it should be possible to connect to it like this:
from openai import OpenAI
client = OpenAI( base_url = "https://api.brev.dev/v1", api_key = [API_KEY] )
completion = client.chat.completions.create( model="nvcf:deepseek/deepseek-r1-distill-qwen-14b:dep-35L0J3gU0hGV3DbiqepYc2MlICG", messages=[{"role":"user","content":"Write a limerick about the wonders of GPU computing as inspired by Brautigan."}], temperature=0.2, top_p=0.7, max_tokens=1024, stream=True )
So I assume this falls under the category of OpenAI-compatible AI providers? When I try to use this provider either in the chat window or with the generate with AI feature I get this error: openai.BadRequestError: json: cannot unmarshal array into Go struct field ChatMessage.messages.content of type string
Here is the full stack trace
ERROR: Exception in ASGI application Traceback (most recent call last): File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/openai/_base_client.py", line 1574, in request response.raise_for_status() File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/httpx/_models.py", line 829, in raise_for_status raise HTTPStatusError(message, request=request, response=self) httpx.HTTPStatusError: Client error '400 Bad Request' for url 'https://api.brev.dev/v1/chat/completions' For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/400
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi result = await app( # type: ignore[func-returns-value] File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in call return await self.app(scope, receive, send) File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/starlette/applications.py", line 107, in call await self.middleware_stack(scope, receive, send) File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in call raise exc File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in call await self.app(scope, receive, _send) File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/marimo/_server/api/auth.py", line 219, in call return await super().call(scope, receive, send) File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/starlette/middleware/sessions.py", line 85, in call await self.app(scope, receive, send_wrapper) File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 191, in call with recv_stream, send_stream, collapse_excgroups(): File "/usr/lib/python3.10/contextlib.py", line 153, in exit self.gen.throw(typ, value, traceback) File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/starlette/_utils.py", line 85, in collapse_excgroups raise exc File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 193, in call response = await self.dispatch_func(request, call_next) File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/marimo/_server/api/middleware.py", line 202, in dispatch return await call_next(request) File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 168, in call_next raise app_exc from app_exc.cause or app_exc.context File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 144, in coro await self.app(scope, receive_or_disconnect, send_no_error) File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/marimo/_server/api/auth.py", line 250, in call await super().call(scope, receive, send) File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/starlette/middleware/authentication.py", line 48, in call await self.app(scope, receive, send) File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/marimo/_server/api/auth.py", line 239, in wrapped_app await app(scope, receive, send) File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/starlette/middleware/cors.py", line 93, in call await self.simple_response(scope, receive, send, request_headers=headers) File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/starlette/middleware/cors.py", line 144, in simple_response await self.app(scope, receive, send) File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/marimo/_server/api/middleware.py", line 174, in call return await self.app(scope, receive, send) File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/marimo/_server/api/middleware.py", line 424, in call await self.app(scope, receive, send) File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/marimo/_server/api/middleware.py", line 424, in call await self.app(scope, receive, send) File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/marimo/_server/api/middleware.py", line 424, in call await self.app(scope, receive, send) [Previous line repeated 1 more time] File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 63, in call await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app raise exc File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app await app(scope, receive, sender) File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/starlette/routing.py", line 716, in call await self.middleware_stack(scope, receive, send) File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/starlette/routing.py", line 736, in app await route.handle(scope, receive, send) File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/starlette/routing.py", line 462, in handle await self.app(scope, receive, send) File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/starlette/routing.py", line 716, in call await self.middleware_stack(scope, receive, send) File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/starlette/routing.py", line 736, in app await route.handle(scope, receive, send) File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/starlette/routing.py", line 290, in handle await self.app(scope, receive, send) File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/starlette/routing.py", line 78, in app await wrap_app_handling_exceptions(app, request)(scope, receive, send) File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app raise exc File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app await app(scope, receive, sender) File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/starlette/routing.py", line 75, in app response = await f(request) File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/marimo/_server/router.py", line 47, in wrapper_func response = await func(request=request) File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/starlette/authentication.py", line 70, in async_wrapper return await func(*args, **kwargs) File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/marimo/_server/api/endpoints/ai.py", line 166, in ai_completion response = await provider.stream_completion( File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/marimo/_server/ai/providers.py", line 590, in stream_completion await client.chat.completions.create(**create_params), File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/openai/resources/chat/completions/completions.py", line 2603, in create return await self._post( File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/openai/_base_client.py", line 1794, in post return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls) File "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/openai/_base_client.py", line 1594, in request raise self._make_status_error_from_response(err.response) from None openai.BadRequestError: json: cannot unmarshal array into Go struct field ChatMessage.messages.content of type string
I don't know if I am doing something wrong here but any support would be much appreciated!
Will you submit a PR?
- [ ] Yes
Environment
{
"marimo": "0.17.7",
"editable": false,
"location": "/home/user001/marimo-test/.venv/lib/python3.10/site-packages/marimo",
"OS": "Linux",
"OS Version": "6.6.87.2-microsoft-standard-WSL2",
"Processor": "x86_64",
"Python Version": "3.10.12",
"Locale": "en_US",
"Binaries": {
"Browser": "--",
"Node": "--"
},
"Dependencies": {
"click": "8.3.0",
"docutils": "0.22.3",
"itsdangerous": "2.2.0",
"jedi": "0.19.2",
"markdown": "3.10",
"narwhals": "2.11.0",
"packaging": "25.0",
"psutil": "7.1.3",
"pygments": "2.19.2",
"pymdown-extensions": "10.16.1",
"pyyaml": "6.0.3",
"starlette": "0.50.0",
"tomlkit": "0.13.3",
"typing-extensions": "4.15.0",
"uvicorn": "0.38.0",
"websockets": "15.0.1"
},
"Optional Dependencies": {
"mcp": "1.21.0",
"openai": "2.7.2",
"polars": "1.35.2",
"pyarrow": "22.0.0"
},
"Experimental Flags": {}
}
Code to reproduce
No response
@ArianJavdan, it sounds like this endpoint is not compatible with the OpenAI API. This is likely coming from sending tools information to their endpoint. Can you reach out to that provider and ask if it is compatible with tools?
@ArianJavdan you can see that the deserialization expect a string for the message content, but a truly open-ai compatible endpoint would support string | ChatMessagePart[] https://platform.openai.com/docs/api-reference/chat/create#chat_create-messages