Tool calls do not work as expected

[d-ivashchuk]

Description

I am trying to call some tools when streaming the response with streamText. It seems like the example below follows the documentation. For some reason I can't access the tool call result in messages:

  const { messages, input, handleInputChange, handleSubmit } = useChat({
    api: "/api/ai/chat-preview",
    initialMessages: [
      {
        id: "xxx",
        role: "assistant",
        content: "Hello! how can I assist you today?",
      },
    ],
  });

  console.log({ messages });

I would expect that whenever there is a choice of SDK to call a tool, the response should end up in messages still. Am I doing something wrong here?

What I ultimately want is that whenever a tool is called and there is no typical chat response route is taken I can take the information from that tool and use it on frontend, hence the need to access it.

Thanks for the support!

Code example

import { openai } from "@ai-sdk/openai";
import { StreamingTextResponse, streamText, tool } from "ai";
import { z } from "zod";
import { db } from "~/server/db";

const jsonSchema = z.object({
  messages: z.array(
    z.object({ role: z.enum(["user", "assistant"]), content: z.string() }),
  ),
  projectId: z.string(),
});

export async function POST(req: Request) {
  const res = await jsonSchema.safeParseAsync(await req.json());

  const result = await streamText({
    model: openai("gpt-4-turbo"),
    system: `You shall only answer abc to any questions from user
    `,
    messages: [...res.data.messages],
    tools: {
      weather: tool({
        description: "Get the weather in a location",
        parameters: z.object({
          location: z.string().describe("The location to get the weather for"),
        }),
        execute: async ({ location }) => ({
          location,
          temperature: 72 + Math.floor(Math.random() * 21) - 10,
        }),
      }),
    },
  });

  return new StreamingTextResponse(result.toAIStream());
}

Additional context

Whenever I try to ask for the weather and console.log() smth in the execute function, it gets triggered correctly. so somehow the end result is not being sent correctly

[lgrammel]

Currently this is not supported. I'm exploring options for adding core tool call and core tool result support to useChat.

[lgrammel]

In-progress PR: https://github.com/vercel/ai/pull/1514 - this will be a larger change, i expect it to take a few days

[d-ivashchuk]

few days seems more than reasonable! I am not 100% bound to this right now, so can wait. Is there maybe a way to achieve what I want in a different way?

@lgrammel thanks a lot for your great work and prompt response, you rock 🚀

[lgrammel]

Thanks! You could try using the legacy providers, but then you'd need to refactor once this lands

[d-ivashchuk]

@lgrammel I saw there was an example of annotations that might be of use here. My use-case is that when there is a specific message response(with some metadata) I want to render that message + some UI. E.g. I would expect something like {role:'user', message:'Provide your email', annotations:{emailInput:true}}

Is this a good use-case for this feature. If so is there a good docs or example of how to do this? I am happy to contribute to docs myself if you give me some basic directions and I manage to figure this out.

I know that maybe RSCs would be a better use case for this whole thing, but I prefer to use useChat() and core in this project.

[lgrammel]

@d-ivashchuk yes, this should be possible. you could define tools without an execute method, handle the tool calls on the client to add information to an array (or alternatively handle the tool calls on the server and forward stream data), and then render the components on the client.

I've added an example of how to show a client component with use-chat in v3.1.2: https://github.com/vercel/ai/pull/1523/files but it will get easier and cleaner with the new feature.

[d-ivashchuk]

@lgrammel thanks a lot for the example! makes sense.

Anecdotally the best docs I could find yesterday was your announcement tweet of streamData feature 🙈

that's what I tried yesterday:

const data = new StreamData();
  const stream = result.toAIStream({
    async onFinal(completion) {
      console.log(completion);
      data.append({
        test: "hello",
      });

      await data.close();
    },
  });
  return new StreamingTextResponse(stream, {}, data);

This is not ideal tbh, as it is missing something. I think I could use messageAnnotation:

async onFinal(completion, message) {
      const jsonCompletion = await completion('Is this message prompting for an email, if yes return {email:true}')
      
      if(jsonCompletion.email){
      data.appendMessageAnnotation({
        email: true,
      });
      }
      

it has a separate call to completions, making it not ideal here. I guess with tools this all would be very much simplified.

[lgrammel]

@d-ivashchuk yes, i think this is a great use case for tools. do you need to call the llm with the tool results, or are they just needed to display a ui component?

[d-ivashchuk]

Might need to call LLM in tool results, but so far for this use case if I get return {role:"tool",{input:"email",message:"Please provide your email"}} it would be more than enough.

Bonus point if message could be streamed as well as an object

[d-ivashchuk]

hey @lgrammel, did you manage to release it in a form that we discussed? I see a related PR merged, but not sure if all the work has been done. Thanks a lot for the answer!

[lgrammel]

@d-ivashchuk i have implemented a slightly different version that focusses on user interactions as client-side tools. I have the sense that you might need something different, e.g. stream data, for your use case. i'm thinking about some additions in that area. would it be sufficient for you if you could attach metadata to an assistant response?

[d-ivashchuk]

well, we can try to figure it out together :D I am thinking that agents could serve as a router of a sort in the chat, e.g. when ai should responed with message + input, when it shall respond with just message and when it shall for example execute some other action based on the whole conversation.

I think agents are designed specifically for that as concept, so in my case not that I need streaming much, even if the response is done not in streamed way I would be fine with that

[d-ivashchuk]

unrelated question to the above but still in the domain of tool calling.

when I call a tool, on the frontend I now have access to the result, but in the toolInvocations. message content is empty. Is there a way to invoke a tool that would respond to user and still execute the tool code @lgrammel?

[animify]

when I call a tool, on the frontend I now have access to the result, but in the toolInvocations. message content is empty. Is there a way to invoke a tool that would respond to user and still execute the tool code @lgrammel?

Running into the same issue with content being empty 💯

[jonoc330]

when I call a tool, on the frontend I now have access to the result, but in the toolInvocations. message content is empty. Is there a way to invoke a tool that would respond to user and still execute the tool code @lgrammel?

Running into the same issue with content being empty 💯

Also having this issue - can't continue a conversation with an assistant as a result

[lgrammel]

@d-ivashchuk @animify @jonoc330 as a first step, I'll add the option to automatically roundtrip to the server when all server tool results are available. The 2nd server call will have the tool results and produce the text if it's set up correctly. Pure server-side roundtrips will come later.

[lgrammel]

feat (ai/react): add experimental_maxAutomaticRoundtrips to useChat: https://github.com/vercel/ai/pull/1681

[zineanteoh]

Pure server-side roundtrips will come later.

look forward to this!

in the meantime, is there a way to accomplish tool calling + streaming final response in client side (without useChat)?

• the current useChat implementation depends on Message which is a shared type between API and UI packages, and i have trouble accomplishing this without useChat

[painpita]

@animify

Running into the same issue with content being empty 💯

Still running into this, did you manage it ?

[myudak]

is this fixed?

[xwchris]

it's still now working

[lgrammel]

maxToolRoundtrips is available for streamText now: https://sdk.vercel.ai/docs/ai-sdk-core/tools-and-tool-calling#example-streamtext

[dunnokiet]

maxToolRoundtrips is available for streamText now: https://sdk.vercel.ai/docs/ai-sdk-core/tools-and-tool-calling#example-streamtext

https://sdk.vercel.ai/docs/guides/rag-chatbot#using-tools

Sir, I followed this dosc guide, but tools still doesn't return any results

[lgrammel]

@dunnokiet most likely a duplicate of https://github.com/vercel/ai/issues/2856 - it'll be fixed in 3.3.22

[joesamcoke]

I'm still experiencing this with sveltekit using the latest sdk and maxToolRoundtrips: 5 on the server and useChat on the client . First I get a message response with empty content with the result of the toolInvocation. isLoading is false until I get the next streamed message response which contains the LLM interpretation of the tool result. isLoading is true throughout the stream of the message. Is this expected behavior?

[jstjoe]

@joesamcoke I think so.

I'm also implementing this in SvelteKit with useChat and what I'm seeing is that the toolInvocations is reactive and updates as the tool executions complete. If you check the toolinvocations.State you can conditionally show the query with a loading state and then reactively render the results as they come in.

I believe 'isLoading' is only an indicator of a an ongoing streaming completion from the LLM. What are you seeing though?

[joesamcoke]

@jstjoe Thanks for the clarification and the heads up on toolinvocations.State.

I'm refactoring this from the legacy function calling and the behaviour seemed a little odd.

Ignore the comment about isLoading - was a issue my end.