Make streaming work with langchain agents

[Royal-lobster]

Is it possible to make streaming work with agents ? I am looking for a solution to stream intermediate steps and answer so users can see the tool selection process from UI along with streaming answer at end.

This is the code i am using on backend

//...imports
export const runtime = "edge";

export const POST = async (req: Request) => {
  const { prompt, debugOptions } = generateInputSchema.parse(await req.json());
  const patchedOptions = { ...config.defaultDebugOptions, ...debugOptions };
  const { stream, handlers } = LangChainStream();

  const model = new ChatOpenAI({
    temperature: 0,
    openAIApiKey: env.OPENAI_API_KEY,
    modelName: "gpt-4-0613",
    callbackManager: CallbackManager.fromHandlers(handlers),
    streaming: true,
  });

  const tools = getTools(patchedOptions);

  const executor = await initializeAgentExecutorWithOptions(tools, model, {
    agentType: "openai-functions",
    maxIterations: 10,
    returnIntermediateSteps: true,
  });

  await executor.call({ input: prompt }).finally(() => {
    handlers.handleChainEnd();
  });
  
  return new StreamingTextResponse(stream);
};

The stream for this is only giving full final answer at the end instead of letter by letter for me with useCompletion hook.

[CarlosZiegler]

Hey, I think this is the same topic as : #80

[Royal-lobster]

Hey, I think this is the same topic as : #80

Oh true, but i think this one is on more integration with langchain agents. i can see the the demo from https://github.com/vercel-labs/ai/issues/80#issuecomment-1593984202 also facing the same issue as mine under functions which is giving out blocking answer at the end instead of streaming effect

[CarlosZiegler]

Yeap, I opened a PR to integrate axios but was closed, here is the explanation:

https://github.com/vercel-labs/ai/pull/140

[Royal-lobster]

Yeap, I opened a PR to integrate axios but was closed, here is the explanation:

#140

I see ! Hopefully we get the update soon. will be waiting for it until then

[Zakinator123]

@Royal-lobster This PR probably solves your issue: #154

[e-roy]

I was able to get streaming working for langchain agents here

[Adebesin-Cell]

I was able to get streaming working for langchain agents here

I ran the code but it doesn't work. I tested both routes news-langchain and news-openai.

[e-roy]

@Adebesin-Cell that's odd, I don't see an error for no api key, the only thing I can think of is maybe it's timing out for you?

I deployed to vercel to test https://openai-functions-with-langchain.vercel.app/ and it seems to work fine with entering "latest on spacex" for an input.

my code could probably improve a bit, I was more focused on getting it to work

[AuYuHui2]

I have encountered the same problem, how can I solve it