Abstract LLM interfaces and Claude 3 Opus

[lukestanley]

Claude 3's Opus may have beat OpenAI's GPT-4's models. I think it would probably be helpful to abstract the interfaces away from just the OpenAI client. Currently the code seems to be tightly coupled to the OpenAI client, and it's unique features. Key parts seem to be main.py, send_chat.py, though I notice model.py does some conditional importing, as does main.py too. base_coder.py also expects a certain completion interface shape, function_call, error classes, statistics and pricing. Anthropic's client has streaming, but not as a keyword argument. They have tool use, but not the same function calling interface. They have system prompts but not a system role. They can list what models are available too, but again that will be different. Vision input is possible but probably different too. The streaming and whole result responses have to be dealt with differently from getting the core result text. Still with retries being needed.

Side note: Openrouter is probably good for some, but others have direct access. Litellm looked promising to me at first (since they seem to offer OpenAI compatible interfaces) but then I found out that they don't convert system messages to system prompts (which causes request errors). Aider is amazing by the way, thank you!

[paul-gauthier]

Yes, I plan to do this.

[SgtPooki]

using aider with LM Studio models would be great

[FreCap]

This is a great idea! would it be possible to have an "easy to configure" adapter for sending requests to corporate Claude/OpenAi gateways?

[paul-gauthier]

For OpenAI gateways you should be able to use existing settings.

Try aider --help to see the available options.

[andr0s]

Claude 3 Opus is imo better than ChatGPT 4, nowadays I use it like 80% of the time, and chatgpt only 20%. So this feature would really be cool to have

[ryanfreckleton]

For what it's worth, litellm seems to be an OpenAI style API wrapper around multiple LLMs, include Claude Opus.

https://github.com/BerriAI/litellm/

[lukestanley]

For what it's worth, litellm seems to be an OpenAI style API wrapper around multiple LLMs, include Claude Opus.

https://github.com/BerriAI/litellm/

I pointed to them in the original message but I found difficulty with their support for system messages via their proxy, but I might have been using it wrong, as I was just using the OpenAI client. Maybe their client would call it as needed? If their server was a drop in replacement, that would be ideal. I actually had a local branch where I was trying it a bit. Other tricky aspects are function calling / tools (which might need to use XML?), and streaming. If I were working on this, I might try to move all the code that uses OpenAI directly into it's own file so it's isolated, then replace each reliant feature, it's got caching, streaming and function calling to deal with, then there is user configuration setting. I wouldn't want to step on anyone's toes, and I prefer not to redo work already in progress, may I ask the status of this? Is there an in progress branch? @paul-gauthier @ryanfreckleton

[paul-gauthier]

I'm likely to use litellm to plug aider into multiple APIs. My only small blocker is that aider uses the OpenAI whisper api, which is not supported by litellm.

There's no branch in progress yet, but this is one of the next features I want to make time for.

[lukestanley]

Good news: LiteLLM added a audio transcription example 3 weeks ago. @paul-gauthier

[paul-gauthier]

Aider now uses litellm to connect to many LLM providers.

https://aider.chat/docs/llms.html

I'm going to close this issue for now, but feel free to add a comment here and I will re-open or file a new issue any time.