Decoding issues with experimental_StreamData

[aaron5670]

Description

The experimental_StreamData function does not work with a Gatsby application.

Code example

Next.js API route

I created the following Next.js endpoint.

import {experimental_StreamData, OpenAIStream, StreamingTextResponse} from 'ai';
import {ChatCompletionMessage} from 'openai/resources/chat/completions';
import {createOpenAIEmbedding} from "@/helpers/createOpenAIEmbedding";
import {pineconeIndex} from "@/helpers/pinecone";
import {openai} from "@/helpers/openai";

export async function POST(req: Request) {
  if (!process.env.OPENAI_API_KEY) {
    return Response.json({error: "OpenAI is not enabled, check your env variables"}, {status: 400});
  }

  try {
    const body = await req.json();
    const messages: ChatCompletionMessage[] = body.messages;

    const messagesTruncated = messages.slice(-6);

    const embedding = await createOpenAIEmbedding(
      messagesTruncated.map((message) => message.content).join("\n"),
    );

    const vectorQueryResponse = await pineconeIndex.query({
      vector: embedding,
      topK: 4,
      includeMetadata: true,
      filter: {type: 'product'},
    });

    console.log('vectorQueryResponse', vectorQueryResponse.matches.map((document) => document.metadata?.title));

    const systemMessage: ChatCompletionMessage = {
      role: "assistant",
      content:
        `
        You are a helpful assistant of the online e-commerce store.
        It is very important that you always answer in French, you must not deviate from this at any cost!
        Use the following bits of context to answer the question at the end.
        Make sure your answer consists of a maximum of about 45 words.
        If you don't know the answer, just say you don't know, don't go making up answers.
        ----------------
        CONTEXT:
        ${vectorQueryResponse.matches
          .map((document) => {
            return `Title: ${document.metadata?.title}\n\nContent:\n${document.metadata?.text}`
          })
          .join("\n\n")}
         `
    };

    const response = await openai.chat.completions.create({
      model: "gpt-3.5-turbo",
      stream: true,
      messages: [systemMessage, ...messagesTruncated],
    });

    // Instantiate the StreamData. It works with all API providers.
    const data = new experimental_StreamData();

    const stream = OpenAIStream(response, {
      experimental_onFunctionCall: async (
        { name, arguments: args },
        createFunctionCallMessages,
      ) => {
        if (name === 'get_current_weather') {
          // Call a weather API here
          const weatherData = {
            temperature: 20,
            unit: args.format === 'celsius' ? 'C' : 'F',
          };

          data.append({
            text: 'Some custom data',
          });

          const newMessages = createFunctionCallMessages(weatherData);
          return openai.chat.completions.create({
            messages: [...messages, ...newMessages],
            stream: true,
            model: 'gpt-3.5-turbo-0613',
          });
        }
      },
      onCompletion(completion) {
        console.log('completion', completion);
      },
      onFinal(completion) {
        // IMPORTANT! you must close StreamData manually or the response will never finish.
        data.close();
      },
      // IMPORTANT! until this is stable, you must explicitly opt in to supporting streamData.
      experimental_streamData: true,
    });

    data.append({
      text: 'Hello, how are you?',
    });

    // IMPORTANT! If you aren't using StreamingTextResponse, you MUST have the `X-Experimental-Stream-Data: 'true'` header
    // in your response so the client uses the correct parsing logic.
    return new StreamingTextResponse(stream, {}, data);
  } catch (error) {
    console.error(error);
    return Response.json({error: "Internal server error"}, {status: 500});
  }
}

export async function OPTIONS() {
  return Response.json({status: "OK"});
}

Next.js (front-end)

In Next.js it works as expected.

"use client";

import {useChat} from "ai/react";

const Page = () => {
  const { data, handleInputChange, handleSubmit, messages } = useChat({
    api: '/api/chat',
  });

  console.log('data', data);
  console.log('messages', messages);

  return (
    <div>
      <form onSubmit={handleSubmit}>
        <input
          type="text"
          name="input"
          onChange={handleInputChange}
        />
        <button type="submit">Send</button>
      </form>
    </div>
  )
}

export default Page

Response of messages in Next.js:

[
    {
        "content": "Hello",
        "role": "user",
        "createdAt": "2024-01-19T09:40:26.742Z",
        "id": "cJCvzYF"
    },
    {
        "id": "4W01c2l",
        "role": "assistant",
        "content": "Bonjour! Comment puis-je vous aider aujourd'hui ?",
        "createdAt": "2024-01-19T09:40:29.129Z"
    }
]

Gatsby (front-end)

And this is my Gatsby front-end code, where I have encoding issues.

import React from 'react';
import { useChat } from 'ai/react';

export const Page = () => {
  const {
    data, handleInputChange, handleSubmit, messages,
  } = useChat({
    api: 'http://localhost:3000/api/chat',
  });

  console.log('data', data);
  console.log('messages', messages);

  return (
    <div>
      <form onSubmit={handleSubmit}>
        <input
          type="text"
          name="input"
          onChange={handleInputChange}
        />
        <button type="submit">Send</button>
      </form>
    </div>
  );
};

Response of messages in Gatsby:

[
    {
        "content": "Hello",
        "role": "user",
        "createdAt": "2024-01-19T09:41:10.737Z",
        "id": "oK7t7Lh"
    },
    {
        "id": "9F2Pg9s",
        "createdAt": "2024-01-19T09:41:12.195Z",
        "content": "0:\"Bonjour\"\n0:\"!\"\n0:\" Comment\"\n0:\" puis\"\n0:\"-\"\n0:\"je\"\n0:\" vous\"\n0:\" aider\"\n0:\" aujourd\"\n0:\"'hui\"\n0:\"?\"\n",
        "role": "assistant"
    }
]

Additional context

No response

[aaron5670]

After some debugging, I found that this is not just a problem with Gatsby. The same problem also occurs when I deploy the Next.js application to Vercel and then connect to the production API via local development.

Then I also get the same encoding/decoding problem.

So I run Next.js locally via npm run dev and the useChat() api goes to a production endpoint.

For example, this Next.js front-end page that I run locally:

"use client";

import {useChat} from "ai/react";

const Page = () => {
  const { data, handleInputChange, handleSubmit, messages } = useChat({
    api: 'https://example-ai-api.vercel.app/api/chat', //  <---- this is the production endpoint
  });

  console.log('data', data);
  console.log('messages', messages);

  return (
    <div>
      <form onSubmit={handleSubmit}>
        <input
          type="text"
          name="input"
          onChange={handleInputChange}
        />
        <button type="submit">Send</button>
      </form>
    </div>
  )
}

export default Page

[antoniocasero]

I'm having the same problem. The data stream appears as an appendix to the message content, as indicated below

2:[{"text":"{\"metadata\":[],\"regularQuery\":{\"error\":null,\"data\":[],\"count\":null,\"status\":200,\"statusText\":\"OK\"}}"}] 
0:"L" 0:"ament" 0:"able" 0:"mente" 0:"," 0:" no" 0:" tengo" 0:" acceso" 0:" a" 0:" documentos" 0:" específ" 0:"icos" 0:" del" 0:" BO" 0:"E" 0:"." 0:" Sin" 0:" embargo" 0:"," 0:" puedo" 0:" ayud" 0:"arte" 0:" a" 0:" encontrar" 0:" la" 0:" información" 0:" que" 0:" neces" 0:"itas" 0:"." 0:" ¿" 0:"En" 0:" qué" 0:" puedo" 0:" as" 0:"ist" 0:"ir" 0:"te" 0:"?"

I'm not sure how to parse this correctly or why the structure is this way. I'm using NextJS and a edge function for calling the OpenAI API. It seems it is related to how OpenAI is giving the response is that right? @MaxLeiter

[aaron5670]

Looks like more people has the same issue: https://github.com/vercel/ai/pull/425#issuecomment-1683732804