Draft: Consolidate External Service Plugins and Refactor code

[jayeshp19]

Proposal: Consolidate External Service Plugins

Currently we have to manage separate plugins for all the external services. This is a draft PR for removing the need to manage all plugins separately. Instead, have all the code in a single plugin without importing extra packages. and it also simplifies the file structure

• current installation

pip install livekit-agents livekit-plugins-openai livekit-plugins-deepgram

• proposed installation

pip install livekit-agents openai

we just need to install live-agents and external model providers as required

try:
    import openai
    from openai.types.chat import ChatCompletionChunk, ChatCompletionMessageParam
    from openai.types.chat.chat_completion_chunk import Choice
except ImportError:
    logger.error("OpenAI API not installed. Please install it using `pip install openai`")
    raise ImportError

• current import method

from livekit.plugins import deepgram, openai, silero

    agent = VoicePipelineAgent(
        vad=silero.VAD.load(),
        stt=deepgram.STT(model=dg_model),
        llm=openai.LLM(),
        tts=openai.TTS(),
        chat_ctx=initial_ctx,
    )

• proposed import method

from livekit.agents.llm import OpenAI
from livekit.agents.stt import DeepGramSTT
from livekit.agents.tts import OpenAITTS
from livekit.agents.vad import Silero

    agent = VoicePipelineAgent(
        vad=Silero.load(),
        stt=DeepgramSTT(model=dg_model),
        llm=OpenAI(),
        tts=OpenAITTS(),
        chat_ctx=initial_ctx,
    )

Proposed File Structure

agents
├── llm
│   ├── openai
│   ├── anthropic
│   └── ... other providers
├── stt
│   ├── deepgram
│   ├── openai
│   ├── google
│   └── ... other providers
├── tts
│   ├── elevenlabs
│   ├── openai
│   └── ... other providers
├── embedder
│   ├── openai
│   └── ... other providers

--- future ideas ---
├── tools
│   ├── slack
│   ├── email
│   ├── crm
│   ├── Tavily (search api)
│   └── ... other tools

with current file structure, we will need to manage plugin separately for each tool we add in future and existing plugins aswell. I think this new structure keeps each service organized within categories and makes future additions easier with just file or two files.

Progress

For the initial MVP, I’ve tested with OpenAI and Silero and below is the implementation progress of current plugins. end-to-end testing is remaining. If we proceed with this proposal, I’ll continue development. Please let me know if you have any questions.

• [x] Openai
• [ ] OpenAI realtime
• [x] Anthropic
• [x] DeepGram
• [x] Elevanlabs
• [x] Google
• [x] Silero
• [x] Azure
• [x] Cartesia
• [ ] Clova
• [ ] llamaindex
• [ ] Browser
• [ ] nltk
• [ ] PlayHT
• [ ] RAG

https://github.com/jayeshp19/agents/tree/refactor-code

@davidzhao @theomonnom I’d love to know your thoughts