gpt turbo patch DEMO

[FergusFettes]

Not a serious PR, just a demo.

Using the normal completions endpoint with Chat models.

With this patch, you can use the summarizing chains and agents with gpt-3.5-turbo and gpt-4.

[colindotfun]

this is cool -- wonder if we can get fake batching behavior in here too, e.g.:

batch_messages = [
    [
        SystemMessage(content="You are a helpful assistant that translates English to French."),
        HumanMessage(content="Translate this sentence from English to French. I love programming.")
    ],
    [
        SystemMessage(content="You are a helpful assistant that translates English to French."),
        HumanMessage(content="Translate this sentence from English to French. I love artificial intelligence.")
    ],
]
result = chat.generate(batch_messages)

from: https://github.com/hwchase17/langchain/blob/a581bce37919b7f0e18c85c4b7043ee92c21364c/docs/modules/chat/getting_started.ipynb#L143

[FergusFettes]

Well yeah, you can do that just with

from langchain.chains.llm import LLMChain
from langchain.llms.openai import OpenAI
from langchain.prompts.prompt import PromptTemplate

llm = OpenAI(temperature=0, model="gpt-3.5-turbo")

template = """
You are a helpful assistant that translates English to French. Translate this sentence from English to French: {text}
"""
prompt = PromptTemplate(input_variables=["text"], template=template)
llm_chain = LLMChain(llm=llm, prompt=prompt)
response = llm_chain.generate(
    [
        {"text": "I love AI"},
        {"text": "I love the ocean"},
    ]
)

for g in response.generations:
    print(g[0].text)

> Je t'aime l'IA
> J'aime l'océan

I think the whole 'ChatCompletions' endpoint was just a smokescreen so nobody noticed they dropped logprobs, I wouldn't pay attention to it. There is no real benefit of eg. the system message atm.

[colindotfun]

yeah, i mean within the MapReduceChain when using turbo, as per: https://github.com/hwchase17/langchain/issues/1643#issuecomment-1474851219

[FergusFettes]

I guess with async and a for loop you could roll your own batching yeah :). Might just mean you would hit rate limits faster though.

[ChrisPF123]

any chance of getting this merged @iltoga @hwchase17

[iltoga]

sorry @ChrisPF123 I don't have privileges to merge pr on this repo

[deedeeharris]

Is this the only solution in order to use turbo for summarization? Or is there an official way?

Thanks.

[baskaryan]

stale, should be using ChatOpenAI for this these days. closing, let me know if i'm missing something!