OpenAI function calling returns invalid JSON when streaming

[jameschetwood]

Description

I'm using OpenAI with Function Calling. I'm not calling multiple functions, I just need to ensure the response is of a certain JSON structure.

If I don't stream the response it works as expected.

Route handler:

import OpenAI from "openai";
import schema from "./schema.json";

export const runtime = "edge";

export async function POST(request: Request) {
    try {
        const openai = new OpenAI({
            apiKey: process.env["OPENAI_API_KEY"],
        });
        const response = await openai.chat.completions.create({
            model: "gpt-4",
            // stream: true,
            messages: [
                {
                    role: "user",
                    content: "Give me 5 questions and answers for a pub quiz",
                },
            ],
            tools: [
                {
                    type: "function",
                    function: {
                        name: "get_questions_and_answers",
                        description: "Get questions and answers",
                        parameters: schema,
                    },
                },
            ],
            tool_choice: {
                type: "function",
                function: { name: "get_questions_and_answers" },
            },
        });
        return Response.json(
            JSON.parse(
                response.choices[0].message.tool_calls?.[0].function.arguments || "",
            ),
        );
    } catch (error) {
        console.error({ error });
        throw new Error("Server error");
    }
}

Response:

{
    "getQuestions": [
        {
            "Question": "What is the capital of Australia?",
            "Answer": "Canberra"
        },
        {
            "Question": "Who wrote 'To Kill a Mockingbird'?",
            "Answer": "Harper Lee"
        },
        {
            "Question": "Who invented the telephone?",
            "Answer": "Alexander Graham Bell"
        },
        {
            "Question": "What is the name of the first manned mission to land on the Moon?",
            "Answer": "Apollo 11"
        },
        {
            "Question": "What is the chemical symbol for Sodium?",
            "Answer": "Na"
        }
    ]
}

I understand that streaming is required to prevent the serverless function from sometimes timing out. When I try to add streaming the response is invalid JSON:

import OpenAI from "openai";
import { OpenAIStream, StreamingTextResponse } from "ai";
import schema from "./schema.json";

export const runtime = "edge";

export async function POST(request: Request) {
    try {
        const openai = new OpenAI({
            apiKey: process.env["OPENAI_API_KEY"],
        });
        const response = await openai.chat.completions.create({
            model: "gpt-4",
            stream: true,
            messages: [
                {
                    role: "user",
                    content: "Give me 5 questions and answers for a pub quiz",
                },
            ],
            tools: [
                {
                    type: "function",
                    function: {
                        name: "get_questions_and_answers",
                        description: "Get questions and answers",
                        parameters: schema,
                    },
                },
            ],
            tool_choice: {
                type: "function",
                function: { name: "get_questions_and_answers" },
            },
        });
        const stream = OpenAIStream(response);
        return new StreamingTextResponse(stream);
    } catch (error) {
        console.error({ error });
        throw new Error("Server error");
    }
}

Response:

{"tool_calls":[ {"id": "call_zyzM9nL9j4x2uojmERkmvcMJ", "type": "function", "function": {"name": "get_questions_and_answers", "arguments": "{\n  \"getQuestions\": [\n    {\n      \"Question\": \"What is the capital of France?\",\n      \"Answer\": \"Paris\"\n    },\n    {\n      \"Question\": \"Who wrote the novel '1984'?\",\n      \"Answer\": \"George Orwell\"\n    },\n    {\n      \"Question\": \"What is the chemical symbol for gold?\",\n      \"Answer\": \"Au\"\n    },\n    {\n      \"Question\": \"Who painted the Mona Lisa?\",\n      \"Answer\": \"Leonardo da Vinci\"\n    },\n    {\n      \"Question\": \"What is the largest planet in our solar system?\",\n      \"Answer\": \"Jupiter\"\n    }\n  ]\n}"}}

I've have a repo which demonstrates the issue: https://github.com/jameschetwood/openai-function-calling

Code example

No response

Additional context

No response

[ozzyonfire]

I think the issue here is that you are trying to parse the accumulated text, but since the text is coming in as chunks, it is not a valid JSON object yet. So the error you are getting is expected.

You can view how the package parses a streamed response by checkout out this file: https://github.com/vercel/ai/blob/main/packages/core/shared/process-chat-stream.ts

[jameschetwood]

I think the issue here is that you are trying to parse the accumulated text, but since the text is coming in as chunks, it is not a valid JSON object yet. So the error you are getting is expected.

You can view how the package parses a streamed response by checkout out this file: https://github.com/vercel/ai/blob/main/packages/core/shared/process-chat-stream.ts

I don't think that's it. If I use a tool like Postman to hit the API (so it's not my code parsing it) I can see the JSON is invalid.

[baptisteArno]

I have the exact same issue but it's only happening on production when deploying on Vercel. 😨

I'm super confused and I don't know how I can debug that...

ai package is up to date.

Here is the invalid JSON it returns:

{"tool_calls":[ {"id": "call_Io6uRXyMRhniyd71oBvAXuG4", "type": "function", "function": {"name": "getWeather", "arguments": "{\n"city": "London"\n}"}}]}

[AaronFriel]

Give the fn-stream library a try. It includes a streaming, incremental JSON parser that returns nodes of the response.

[suryasanchez]

Same issue here after migrating from function_call to tool_choice. The JSON is always missing "]}" at the end.

The fix I did is simply adding the ]} after getting all the stream 😅

EDIT:

I noticed that the bug doesn't appear when setting : tool_choice: "auto"

EDIT 2:

Probably the parser is not correctly done for tool_choice: { type: "function", function: { name: "function_name" } }

When stream: false here are the two outputs:

tool_choice: "auto"

{
  id: 'chatcmpl-96tpzV74JnwXfuHGy106kc8Ai3swp',
  object: 'chat.completion',
  created: 1711431635,
  model: 'gpt-4-0125-preview',
  choices: [
    {
      index: 0,
      message: {
        role: 'assistant',
        content: null,
        tool_calls: [
          {
            id: 'call_OkHES6z9H2ImxtkuaAp1UmnU',
            type: 'function',
            function: {
              name: 'get_current_weather',
              arguments: '{"location":"Paris, France","unit":"celsius"}'
            }
          }
        ]
      },
      logprobs: null,
      finish_reason: 'tool_calls'
    }
  ],
  usage: {
    prompt_tokens: 82,
    completion_tokens: 22,
    total_tokens: 104
  },
  system_fingerprint: 'fp_a7daf7c51e'
}

tool_choice: { type: "function", function: { name: "function_name" } }

{
  id: 'chatcmpl-96tr4X6WyqtNHfQCHWqYKPf2HwCPU',
  object: 'chat.completion',
  created: 1711431702,
  model: 'gpt-4-0125-preview',
  choices: [
    {
      index: 0,
      message: {
        role: 'assistant',
        content: null,
        tool_calls: [
          {
            id: 'call_eJ1yXrhRbTSCMUAaOuJ4nK3G',
            type: 'function',
            function: {
              name: 'get_current_weather',
              arguments: '{"location":"Paris, France","unit":"celsius"}'
            }
          }
        ]
      },
      logprobs: null,
      finish_reason: 'stop'
    }
  ],
  usage: {
    prompt_tokens: 92,
    completion_tokens: 12,
    total_tokens: 104
  },
  system_fingerprint: 'fp_a7daf7c51e'
}

So the only difference I see is around finish_reason: 'stop' if that helps to debug.

[lgrammel]

In v3.0.14, we added 2 experimental functions for the object generation use case:

• experimental_generateObject
• experimental_streamObject

You can find examples here: https://github.com/vercel/ai/tree/main/examples/ai-core/src

[suryasanchez]

In v3.0.14, we added 2 experimental functions for the object generation use case:

• experimental_generateObject
• experimental_streamObject

You can find examples here: https://github.com/vercel/ai/tree/main/examples/ai-core/src

Thank @lgrammel that worked great!

The only thing is that OpenAIStream doesn't work with experimental_streamObject in api route (app router) so I wrote my own stream like

const stream = new ReadableStream({
    async start(controller) {
      for await (const partialObject of result.partialObjectStream) {
        const chunk = JSON.stringify(partialObject) + "\n"; // Format each chunk
        controller.enqueue(new TextEncoder().encode(chunk));
      }
      controller.close();
    },
  });

  return new StreamingTextResponse(stream, {
    headers: { "Content-Type": "application/json" },
  });

And then in the UI I do something like below. Note that I don't use chat, simply call the AI via API route.

let buffer = ""; // String buffer to accumulate incoming data

const response = await fetch(`/api/chat/`, {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ message }),
});

  const reader = response.body.getReader();

const readStream = async () => {
  while (true) {
    const { done, value } = await reader.read();
    if (done) {
      return; // Resolve the promise when the stream is finished
    }

    const chunk = new TextDecoder().decode(value);
    buffer += chunk; // Accumulate the chunk into the buffer

    let boundary;
    while ((boundary = buffer.indexOf("\n")) !== -1) {
      const json = buffer.substring(0, boundary);
      buffer = buffer.substring(boundary + 1);

      try {
        const data = JSON.parse(json);
        console.log(data);

      } catch (error) {
        console.error("Error parsing JSON object", error);
      }
    }
  }
};

await readStream();

I'm pretty sure there is a better way but hope this will help someone.

[lgrammel]

@suryasanchez great example! I'll think about how we can integrate something along those lines.

Here is an example of using streamObject with RSCs for incremental rendering:

https://github.com/vercel/ai/tree/main/examples/next-ai-rsc/app/stream-object