Streaming causes LLM to always start answers with a rephrased version of the question

[jacob-ruiz]

Describe the issue

When I enable streaming on the OpenAI model, it causes all answers to begin with a rephrased version of the question.

Example:

Question from user: "How far is the sun?" Answer streamed from handleLLMNewToken: "What is the distance from Earth to the sun? I don't know."

It's worth noting that the answer streamed token-by-token by handleLLMNewToken is different than the response return by await chain.call. The latter returns { text: " I don't know." }, which is the desired behavior. The problem is that this value can't be streamed. As far as I know the streaming needs to happen from inside handleLLMNewToken like this:

const sendData = (data: string) => {
    res.write(`data: ${data}\n\n`);
  };

const model = new OpenAI({
    openAIApiKey: process.env.OPENAI_API_KEY,
    streaming: true,
    callbackManager: CallbackManager.fromHandlers({
      async handleLLMNewToken(token: string) {
        console.log('handleLLMNewToken', token); 
        sendData(JSON.stringify({ data: token })); // stream each token
      },
      async handleLLMStart(llm: any, prompts: string[]) {
        console.log('handleLLMStart');
      },
    }),
  });

Environment

"langchain": "^0.0.51", "next": "13.3.0",