dspy [Request] - Adding functionality of other commercial LLMs ?

Apart from the commercial LLMs that are available in the module, are you guys working on introducing other models such as Claude, Bard or similar. If that's something on the pipeline, I would like to contribute.

Jan 14 '24 04:01 yash-srivastava19

Hey Yash! We want to add support for LangChain LLMs out of the box, so you can import any of their LLMs and use them in DSPy. We are just starting to look into what’s needed for that

Jan 14 '24 04:01 okhat

Sure, if there's something where I can be of some utility, hit me up.

Jan 14 '24 05:01 yash-srivastava19

Adding support for more llms, both commercial and open source, can be handled either on case by case basis or in a more scalable way with a good abstraction. I'm not sure about the downstream implications but perhaps the inspiration from llama_index's ServiceContext may be helpful (https://docs.llamaindex.ai/en/stable/module_guides/models/llms/usage_custom.html). Btw, it also supports Langchain llms

It'd be fantastic if we could use their llm abstractions, like that:

Langchain llm:

from langchain.llms import OpenAI
from llama_index.llms import LangChainLLM

model = "gpt-3.5-turbo"
llm_llamaindex = LangChainLLM(llm=OpenAI())
resp = llm_llamaindex.complete("what is colBERT?")
print(resp)

dspy.settings.configure(lm=llm_llamaindex)

Llama2 via TogehterAI:

from llama_index.llms import TogetherLLM

model = "meta-llama/Llama-2-70b-chat-hf"
llm_llamaindex = TogetherLLM(
     model=model, api_key= os.getenv("TOGETHERAI_API_KEY"),
 )
resp = llm_llamaindex.complete("what is colBERT?")
print(resp)
dspy.settings.configure(lm=llm_llamaindex)

Jan 18 '24 18:01 ukituki

Adding support for more llms, both commercial and open source, can be handled either on case by case basis or in a more scalable way with a good abstraction. I'm not sure about the downstream implications but perhaps the inspiration from llama_index's ServiceContext may be helpful (https://docs.llamaindex.ai/en/stable/module_guides/models/llms/usage_custom.html). Btw, it also supports Langchain llms

It'd be fantastic if we could use their llm abstractions, like that:

Langchain llm:
from langchain.llms import OpenAI
from llama_index.llms import LangChainLLM

model = "gpt-3.5-turbo"
llm_llamaindex = LangChainLLM(llm=OpenAI())
resp = llm_llamaindex.complete("what is colBERT?")
print(resp)

dspy.settings.configure(lm=llm_llamaindex)
Llama2 via TogehterAI:
from llama_index.llms import TogetherLLM

model = "meta-llama/Llama-2-70b-chat-hf"
llm_llamaindex = TogetherLLM(
     model=model, api_key= os.getenv("TOGETHERAI_API_KEY"),
 )
resp = llm_llamaindex.complete("what is colBERT?")
print(resp)
dspy.settings.configure(lm=llm_llamaindex)

Good abstractions, as that of Langchain would be really good for scaling. I will read through the ServiceContext and then we can we see how to take it from there. If you have any suggestions, or want to work together on this, we can do that !

Jan 19 '24 04:01 yash-srivastava19

Many commercial providers work out of the box with the OpenAI library though. For example:

mixtral = dspy.OpenAI(
    model="mistralai/Mixtral-8x7B-Instruct-v0.1",
    max_tokens=250,
    model_type="chat",
    api_key=os.environ["TOGETHER_API_KEY"],
    api_base="https://api.together.xyz/v1",
)

dspy.settings.configure(lm=mixtral)

Jan 21 '24 17:01 stillmatic

Sorry if I'm misunderstanding but does adding support for other LLMs mean supporting chat APIs? (referring to this issue Because at the moment the way prompts are built is very much tailored for instruct models.