Support multiple LLMs through Litellm

[dhirenmathur]

Replace LangChain Chat Clients with LiteLLM

Objective

Replace all LangChain chat client implementations with LiteLLM while maintaining existing functionality. APIs to manage user's LLM choice and keys and ensure that right LLM is selected during processing and agent execution.

Requirements

• Keep existing provider service pattern intact
• Support all current LLMs through LiteLLM
• Maintain inference and agent interaction functionality
• Remove CrewAI specialized handling (This was added because crewai was not able to support anthropic through the langchain client due to some conflict witht their internal litellm)
• Preserve streaming functionality

Implementation

1. Replace LangChain imports with LiteLLM
2. Update provider services
3. Remove CrewAI-client specific code
4. Update configurations and environment variables

Testing

• Verify provider service functionality
• Test all LLM integrations
• Validate inference and agent operations
• Check streaming implementations
• Run existing test suite

Success Criteria

• [ ] All LangChain chat clients replaced
• [ ] Potpie supports any model that is supported by Litellm
• [ ] User is able to set preference and keys for their LLM of choice
• [ ] No breaks in existing functionality
• [ ] Streaming working as expected

[dhirenmathur]

Need to think about how to select the correct provider, model and key

[dhirenmathur]

/bounty 10

[algora-pbc[bot]]

💎 $50 bounty • potpie.ai

Steps to solve:

1. Start working: Comment /attempt #222 with your implementation plan
2. Submit work: Create a pull request including /claim #222 in the PR body to claim the bounty
3. Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

Thank you for contributing to potpie-ai/potpie!

Add a bounty • Share on socials

Attempt	Started (GMT+0)	Solution
🔴 @dfordp	Jan 14, 2025, 12:44:55 PM	WIP
🟢 @PRANJALRANA11	Jan 20, 2025, 3:04:12 PM	WIP

[dhirenmathur]

/bounty 50

[dfordp]

/attempt #222

Options

• Cancel my attempt

[PRANJALRANA11]

/attempt #222

Options

• Cancel my attempt

[PRANJALRANA11]

@dhirenmathur does we have to support all llm with litellm or the llm which are present in current application for those just we have to change from langchain to lite

[PRANJALRANA11]

and also does i have to completely remove langchain in all aspects like - prompttemplates , tool calls or just the chat

[dhirenmathur]

hey @PRANJALRANA11 we want to incorporate litellm across the app so that we can support all LLMs. We use instances of langchain client and the crewai client, this makes the provider_service and agent code really messy, ideally we want to replace both with a single litellm instance.

Yes if we are not going to use the langchain client then it does not make sense to keep the prompt template etc.

For tools, I think we can continue with langchain tools, what do you think?

[dhirenmathur]

Hey @dfordp , Thanks for attempting this! In the interest of time, I'm assigning this to @PRANJALRANA11 since there have been no follow up messages in ~10 days. We will have other open issues for you to contribute on when you're ready!

[dfordp]

@dhirenmathur sorry I wasn't able to do it, I was 50% through to doing it but got busy with work.

[psood708]

@dhirenmathur is this up for takes or still assigned?

[PRANJALRANA11]

Hey @dhirenmathur could you please assign this issue to someone else

[MihirsinhChauhan]

@dhirenmathur I would like work on this issue.

[dhirenmathur]

Hi We will be taking this issue up internally in the interest of time, thanks for the offers folks, we are opening new issues soon that you will be able to contribute to! @MihirsinhChauhan @psood708 looking forward to your contributions!