Tool usage in ChatGoogleGenerativeAI is totally broken and so are the tests for it

[Pupix]

Checked other resources

• [X] I added a very descriptive title to this issue.
• [X] I searched the LangChain.js documentation with the integrated search.
• [X] I used the GitHub search to find a similar question and didn't find it.
• [X] I am sure that this is a bug in LangChain.js rather than my code.
• [X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

Any of the tutorials that use tools will do.

Error Message and Stack Trace (if applicable)

No response

Description

While doing the tutorial about the weather tool, I noticed that the AI would keep responding as if I was the one telling it the weather and not the other way around.

The code seems to early exit if there's a BaseMessage with string contents, without checking if it's from a tool or not.

It can be seen here https://github.com/langchain-ai/langchainjs/blob/eb9ce8d05ab5bc5fc07b24c33e651943149c07fe/libs/langchain-google-genai/src/utils/common.ts#L86-L92

There are tool checks a few lines down, but it will never reach if the tool just responds with 'It's 60 degrees.'

More than that, looking at the message formatting, the tool responses must be objects and not strings like they are sent here. https://github.com/langchain-ai/langchainjs/blob/eb9ce8d05ab5bc5fc07b24c33e651943149c07fe/libs/langchain-google-genai/src/utils/common.ts#L110-L114

I tried returning objects from the tool, but it stringified them somewhere.

The typedef for FunctionResponse

From: https://github.com/google-gemini/generative-ai-js/blob/cec7440c0d34d29298333d920d4f44f8478d27d5/types/content.ts#L160

The tests are also broken. They run as if the users tells the weather or whatever to the AI instead of properly using functionResponse. https://github.com/langchain-ai/langchainjs/blob/eb9ce8d05ab5bc5fc07b24c33e651943149c07fe/libs/langchain-google-genai/src/tests/chat_models.test.ts#L184-L201

As a workaround I monkeypatched langchain by moving the above mentioned string check at the bottom of the if/else if block

and wrapped response in functionResponse in an object.

Probably not the best way to handle things, but it works.

System Info

Windows Node 22.x.x yarn 1.x.x

[dosubot[bot]]

Hey @Pupix! 👋 I'm here to help you with any bugs, questions, or contributions you have in mind. Let's squash those bugs together!

I found a similar unsolved discussion that might be relevant to your issue: langchain agent -> Exiting LLM run with output: { "generations": [ [ null ] ] } ^[1]. Additionally, there was a closed issue related to providing functionResponses, which suggested ensuring that the values of input variables are strings before invoking methods ^[2].

To address the problem where tool responses are incorrectly processed as user messages, you need to ensure that tool responses are structured as objects using the FunctionResponse interface. The LangChain.js codebase should correctly parse tool responses and create an AIMessage with the appropriate content. Here's a snippet that demonstrates how to handle tool responses correctly:

const chatGenerationContent = chatResult.generations[0].message.content;
if (typeof chatGenerationContent !== "string") {
  throw new Error("OllamaFunctions does not support non-string output.");
}
let parsedChatResult;
try {
  parsedChatResult = JSON.parse(chatGenerationContent);
} catch (e) {
  throw new Error(
    `"${this.llm.model}" did not respond with valid JSON. Please try again.`
  );
}
const calledToolName = parsedChatResult.tool;
const calledToolArguments = parsedChatResult.tool_input;
const calledTool = functions.find((fn) => fn.name === calledToolName);
if (calledTool === undefined) {
  throw new Error(
    `Failed to parse a function call from ${this.llm.model} output: ${chatGenerationContent}`
  );
}
if (calledTool.name === this.defaultResponseFunction.name) {
  return {
    generations: [
      {
        message: new AIMessage({
          content: calledToolArguments.response,
        }),
        text: calledToolArguments.response,
      },
    ],
  };
}

const responseMessageWithFunctions = new AIMessage({
  content: "",
  additional_kwargs: {
    function_call: {
      name: calledToolName,
      arguments: calledToolArguments
        ? JSON.stringify(calledToolArguments)
        : "",
    },
  },
});

return {
  generations: [{ message: responseMessageWithFunctions, text: "" }],
};

This code ensures that tool responses are parsed and structured correctly, preventing them from being misinterpreted as user messages ^[3][4].

To continue talking to Dosu, mention @dosu.

[BaharChidem]

Hi there! We are a group of 3 students from the University of Toronto and we are very interested in fixing this issue and also adding some tests. We will submit a PR for this issue by end of November.

[dosubot[bot]]

Hi, @Pupix. I'm Dosu, and I'm helping the LangChain JS team manage their backlog. I'm marking this issue as stale.

Issue Summary:

• You reported a bug where AI responses are processed incorrectly due to improper handling of BaseMessage with string contents.
• I suggested using the FunctionResponse interface to structure tool responses and provided a code snippet.
• BaharChidem and two other students from the University of Toronto are interested in fixing the issue and plan to submit a pull request by the end of November.

Next Steps:

• Please let us know if this issue is still relevant to the latest version of the LangChain JS repository by commenting on this issue.
• If there is no further activity, this issue will be automatically closed in 7 days.

Thank you for your understanding and contribution!

[Pupix]

I looked up the tests and they are still broken. If I had to guess the rest of it is broken as well, but haven't used langchain.js since I can't due to the issues I highlighted above.

[Zeryther]

I've been trying for the past two days to get Langchain to work with Google Generative AI. The documentation and tools for it are in a very rough state.

@Pupix do you mind sharing what you've been using instead?

[Pupix]

@Zeryther I poked around the code and rolled my own solution based on what langchain does. From what I gathered there are just a handful of functions that do actual work, the rest is glue and utilities to handle different LLMs and whatnot. Another problem I found with langchain is that token streaming is not supported. So if you care about latency and getting the responses from LLM ASAP this tool ain't it, but it's great to create complex workflows that will come to an end eventually.

For my use case I needed to have TTS and sub-second responses, thus needed to do sentence splitting as soon as tokens start coming in instead of waiting for the whole LLM response, which is what this library does, or at least used to do when I last looked at it.