I can't use two models with `dspy.OpenAI`

[Su3h7aM]

When I try to use two different models using dspy.OpenAI, it ends up using only the last declaration. As in the example below, even using llama3 it ends up using phi3. In this case I am using dspy.OpenAI but behind the scenes there are two instances of llama.cpp (It does not appear to be a problem with llama.cpp, as using the API directly the problem does not recur). When I try to use dspy.OpenAI for Llama-3 and dspy.OllamaLocal for Phi-3, everything works correctly.

In short: Even using two models, the responses are only generated by the last declared model.

import dspy

llama3 = dspy.OpenAI(api_base="http://localhost:8080/v1/", model_type="chat", api_key="no-key")
phi3 = dspy.OpenAI(api_base="http://localhost:4040/v1/", model_type="chat", api_key="no-key")
#phi3 = dspy.OllamaLocal(model="phi3")

#dspy.settings.configure(lm=llama3)

question = "Who are you? Who developed you?"

class QA(dspy.Signature):
    question: str = dspy.InputField()
    answer: str = dspy.OutputField()

with dspy.context(lm=llama3):
    a = dspy.TypedPredictor(QA)(question=question)

    print("\nLlama3: ")
    #print(llama3(question))
    print(a.answer)

with dspy.context(lm=phi3):
    b = dspy.TypedPredictor(QA)(question=question)

    print("\nPhi3: ")
    #print(phi3(question))
    print(b.answer)

#llama3.inspect_history(n=1)
#phi3.inspect_history(n=1)

Env: DSP_CACHEBOOL=false

DSPy: v2.4.5 OpenAI: v1.23.4 Python: v3.11.9 OS: Fedora Linux 40 (KDE Plasma), Linux 6.8.7-300.fc40.x86_64 CPU: AMD Ryzen 7 5700X 8-Core Processor RAM: 32010 Mi

[arnavsinghvi11]

Hi @Su3h7aM , #744 is related but not mergeable yet.

the responses are only generated by the last declared model.

this is bit unclear. Are the generations only from phi3, or does the inspect_history printed value only show phi3 generations? Specifying the model within the context manager ensures the corresponding generations are from the specified model.

[Su3h7aM]

this is bit unclear.

Sorry, I'll try to be more specific.

---

llama3 = dspy.OpenAI(api_base="http://localhost:8080/v1/", model_type="chat", api_key="no-key")
phi3 = dspy.OpenAI(api_base="http://localhost:4040/v1/", model_type="chat", api_key="no-key")

Here for example, all requests go to http://localhost:4040/v1/ even when using with dspy.context(lm=llama3): which should send to http://localhost:8080/v1/

---

phi3 = dspy.OpenAI(api_base="http://localhost:4040/v1/", model_type="chat", api_key="no-key")
llama3 = dspy.OpenAI(api_base="http://localhost:8080/v1/", model_type="chat", api_key="no-key")

Here they all go to http://localhost:8080/v1/

[Su3h7aM]

Hi @Su3h7aM , #744 is related but not mergeable yet.

I just tested it on his branch and it worked as expected. Thanks

[sean-hacker]

Are there any updates on this? Would love to be able to use both GPT-3.5 and GPT-4 within a single DSPy program.