Enable support for other LLMs

[TechNickAI]

Claude 2 looks interesting

[hanselke]

https://github.com/deep-diver/LLM-As-Chatbot looks interesting. in particular it's https://github.com/deep-diver/PingPong mechanism to isolate the prompt difference requirements.

[TechNickAI]

Note that if you use the openrouter.ai option, you can choose any model they support.

https://openrouter.ai/docs

To do so, add your openrouter_api_key to the config file at $HOME/.aicodebot.yaml

openrouter_api_key: sk-or-v1....

You can then specify which model to use, you can pass AICODEBOT_MODEL as an environment variable.

I'll work on adding that as a config option as well.

[hanselke]

Few questions:

1. how are you planning to support locally run LLMs? have an ugly hack here https://github.com/hanselke/AICodeBot/commit/ea7ccae9ef07b58c645549490447849c2e9fb76c

Think thats the wrong approach tho, to run the LLMs within the core AICodebot code.

I would suggest using nats as a queue/service discovery mechanism, like https://github.com/openbnet/troybot/blob/main/services/stt/nats_stt.py

Benefits of using nats

• locally run GPUs that dont have to be on the development machine
• multiple workers in the queue so that you can run seperate jobs in parallel, or as a HA fallback.

Could do a docker compose to get the LLM + nats running.

2. we're gona need some sort of prompt management mechanism for different LLMs. not sure how you want to approach it.

Using the default prompts, falcon-7b is totally off the mark. https://github.com/hanselke/AICodeBot/commit/5e09d69a3400d8cc36240e129b8aab6867fa7d03 https://github.com/hanselke/AICodeBot/commit/a61f8b6fefaf6ee42b657f45c6842647cec957a6

seems like its giving us back the inputs.

[TechNickAI]

I understand and appreciate the importance of this now.

@hanselke I just re-factored the layout of the commands, so your current code isn't compatible -

I'll look into adding support for other LLMS soon!

[TechNickAI]

Disappointing results - nothing is as good as ChatGPT-4 so far

[hanselke]

Yeap. Was gonna try with https://github.com/bigscience-workshop/petals to see if that works.

Have work this week I need to get done tho, hopefully I’ll be able to find time next week.

On 14 Aug 2023, at 2:01 AM, Nick Sullivan @.***> wrote:

Disappointing results - nothing is as good as ChatGPT-4 so far

— Reply to this email directly, view it on GitHub https://github.com/gorillamania/AICodeBot/issues/36#issuecomment-1676425482, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMJWFIBP24KOQD4HSWRYRDXVEI7FANCNFSM6AAAAAA2HU6S24. You are receiving this because you were mentioned.

[ishaan-jaff]

@hanselke @gorillamania i'm the maintainer of liteLLM - https://github.com/BerriAI/litellm/ We allow you to call OpenAI, Azure, TogetherAI, Hugging Face, Claude, Cohere (and 50+ LLMs) using OpenAI chatGPT inputs/outputs.

We built liteLLM to solve the problem mentioned in this issue

Here's how litellm calls work

from litellm import completion

## set ENV variables
os.environ["OPENAI_API_KEY"] = "openai key"
os.environ["COHERE_API_KEY"] = "cohere key"

messages = [{ "content": "Hello, how are you?","role": "user"}]

# openai call
response = completion(model="gpt-3.5-turbo", messages=messages)

# cohere call
response = completion("command-nightly", messages)

If this is useful for this issue - I'd love to make a PR to help gorillamania (I noticed your using ChatOpenAI we have a langchain integration using ChatLiteLLM if you'd prefer staying with langchain