pipecat-ai
LiteLLM SDK LLM Service
https://github.com/BerriAI/litellm
Rather than having a ton of individual services with different formats, using the LiteLLM python SDK means changing models is literally as simple as changing a single line without down-stream issues (e.g. different formats for functions)
This is an interesting idea. I don't think we want to replace all of the existing LLM services right away, as the devil is often in the details with abstraction layers like this. But we could definitely build a LiteLLMService that sits alongside existing services, and use that as a proving ground of sorts.
Yes definitely, I think just as an option solves it for us.
The problem is not the different LLm processors, it all the different formats they expect
Yes, this will be extremely useful for us. We are deploying newer open source model with vllm, which litellm exposes as OpenAI compatible APIs. Having Litellm integration (in pipecat and pipecat flows) will allow us to switch models and test easily.
Need this for Spend Tracking https://docs.litellm.ai/docs/proxy/cost_tracking
Is it something that is going to enter in roadmap? It would be very interesting, especially to add some routing and fallback capacity based on extensive strategy (errors, latency, token consumption)
We are definitely interested in this but time hasn't permitted just yet. If anyone is interested, we're happy to collaborate on this.
@markbackman I am thinking to start with this as I am already familiar with LiteLLM. Just checking If your team is already working on this?
@shubhamofbce we aren't working on this. LLMs in Pipecat are quite complex and require a good understanding of how Pipecat works. If you decide to work on this, please look at other LLM services and the context aggregators as a reference.