livekit
feat: add Cohere plugin for LiveKit Agents
Hello, and thank you for your PR! In most cases, if the API is quite similar to OpenAI's, we push new providers as additional function to OpenAI (e.g., recent PRs regarding OpenRouter or OVH) plugin. Could you update it to just be part of openai plugin?
Hello, and thank you for your PR! In most cases, if the API is quite similar to OpenAI's, we push new providers as additional function to OpenAI (e.g., recent PRs regarding OpenRouter or OVH) plugin. Could you update it to just be part of openai plugin?
Hi @Hormold thanks for your message! I will. 😄
Hey! Tested the Cohere integration and found two issues that need fixing before this can work properly with voice agents.
First, Cohere API returns 400: message must be at least 1 token long when there's no user message in the chat context. This happens in voice agents when generate_reply(instructions="...") is called without user input (like in on_enter() to greet the user). OpenAI handles this fine but Cohere doesn't... looks like Cohere requires at least one user message to generate a response.
Second, tool calling breaks with 400: schema 'type' must be a string. Array 'type' is unsupported for this model. Cohere's OpenAI-compatible API seems to have stricter JSON schema requirements than OpenAI - it doesn't accept union/array types that LiveKit generates for function tools. Need to figure out what schema format Cohere actually expects and adapt the tool serialization.
I took a look at the first issue. One idea: we could add a check in the chat method (Cohere-only) to see if there’s at least one user message in the context. If not, we either auto-inject a small placeholder user message (e.g. “Hello”) so Cohere is happy, or just throw an exception instead. Not sure which direction makes more sense here, but this would at least guarantee we don’t hit that 400 on empty contexts.
I dug into the second issue as well. Ended up fixing it by setting _strict_tool_schema=False for Cohere.
I took a look at the first issue. One idea: we could add a check in the chat method (Cohere-only) to see if there’s at least one user message in the context. If not, we either auto-inject a small placeholder user message (e.g. “Hello”) so Cohere is happy, or just throw an exception instead. Not sure which direction makes more sense here, but this would at least guarantee we don’t hit that 400 on empty contexts.
@Hormold can you confirm?
Adding a placeholder message should be fine. This is what we did with Gemini Realtime:
Hey, I tested, and the PR looks good. One minor thing is a conflict here. Also, I encountered a couple of timeouts on Cohere responses.
Hey, I tested, and the PR looks good. One minor thing is a conflict here. Also, I encountered a couple of timeouts on Cohere responses.
Thanks for testing! Regarding the timeouts - Cohere API can have high latency (25+ seconds TTFT) which may cause timeout errors in real-time applications. Consider:
- For a small, fast model:
command-r7b-12-2024 - For general purpose use:
command-r-08-2024 - For more advanced capabilities:
command-a-03-2025 - Increasing timeout values for production use
- Setting appropriate max_completion_tokens to reduce response time
I also updated the latest Cohere Command models (text generation).
Thanks! Works great!
Thanks for the feedback! Glad it's working well. Any final changes needed or ready to merge? 🚀