Google Gen AI Structured Output Error

[tonyabracadabra]

Checked other resources

• [X] I added a very descriptive title to this issue.
• [X] I searched the LangChain.js documentation with the integrated search.
• [X] I used the GitHub search to find a similar question and didn't find it.
• [X] I am sure that this is a bug in LangChain.js rather than my code.
• [X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

import { ChatGoogleGenerativeAI } from "@langchain/google-genai";

import { z, ZodTypeAny } from "zod";

interface GenerateStructuredOutputParams<T extends ZodTypeAny> {
  schema: T;
  request: string;
  temperature?: number;
}

export async function generateStructuredOutput<T extends ZodTypeAny>({
  schema,
  request,
  temperature = 1.5,
}: GenerateStructuredOutputParams<T>): Promise<z.infer<T>> {
  const model = new ChatGoogleGenerativeAI({
    model: "gemini-1.5-flash",
    temperature,
  });

  const structuredLlm = model.withStructuredOutput(schema);
  return await structuredLlm.invoke(request);
}

Error Message and Stack Trace (if applicable)

⨯ node_modules/@langchain/google-genai/dist/output_parsers.js (65:1) @ GoogleGenerativeAIToolsOutputParser.parseResult
 ⨯ Error: No parseable tool calls provided to GoogleGenerativeAIToolsOutputParser.
    at async generateStructuredOutput (./src/server/lib/ai/llm.ts:13:12)
    at async generateSingleScript (./src/server/actions/product/script.ts:27:20)
    at async Promise.all (index 1)
    at async generateProductScripts (./src/server/actions/product/script.ts:43:21)
digest: "672233516"
  63 |         });
  64 |         if (tools[0] === undefined) {
> 65 |             throw new Error("No parseable tool calls provided to GoogleGenerativeAIToolsOutputParser.");
     | ^
  66 |         }
  67 |         const [tool] = tools;
  68 |         const validatedResult = await this._validateResult(tool.args);
 POST /dashboard 500 in 3059ms

Description

When attempting to generate structured output using Google's Generative AI (Gemini) through LangChain, an error occurs: CopyError: No parseable tool calls provided to GoogleGenerativeAIToolsOutputParser. Context

The error occurs when using the ChatGoogleGenerativeAI class from @langchain/google-genai. The code is attempting to use structured output with a Zod schema. The model being used is "gemini-1.5-flash".

System Info

Operating System: Platform: darwin Arch: arm64 Version: Darwin Kernel Version 23.5.0: Wed May 1 20:17:33 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T6031 Available memory (MB): 98304 Available CPU cores: 14 Binaries: Node: 20.11.1 npm: 10.2.4 Yarn: N/A pnpm: 9.6.0 Relevant Packages: next: 15.0.0-canary.160 // There is a newer canary version (15.0.0-canary.173) available, please upgrade! eslint-config-next: 15.0.0-canary.69 react: 19.0.0-rc-778e1ed2-20240926 react-dom: 19.0.0-rc-778e1ed2-20240926 typescript: 5.6.2 Next.js Config: output: N/A

[dosubot[bot]]

I found a similar issue that might be relevant to your problem: No tools_call in message error on ChatVertexAI. This issue was closed, and the solution involved ensuring that the input message and schema were correctly aligned with the expected output from the model ^[1].

To resolve the "No parseable tool calls provided to GoogleGenerativeAIToolsOutputParser" error, ensure that the generations array passed to the parseResult method contains messages with a tool_calls property that is an array. Here is an example of how the generations array should be structured:

const generations = [
  {
    message: {
      tool_calls: [
        {
          args: {
            // your tool call arguments here
          }
        }
      ]
    }
  }
];

Make sure your input data is correctly formatted to include the necessary tool_calls array ^[2].

To continue talking to Dosu, mention @dosu.

[jacoblee93]

Does this persist with larger models? What if you prompt more strictly to include something like "you must always use tools if available"?

Unfortunately tool calling/structured output isn't perfect in all cases - CC @bracesproul to make sure we're doing all we can here.

[tonyabracadabra]

Does this persist with larger models? What if you prompt more strictly to include something like "you must always use tools if available"?

Unfortunately tool calling/structured output isn't perfect in all cases - CC @bracesproul to make sure we're doing all we can here.

I don't understand, I am just using it for structured output without requiring it to use any tools.

[jeloi]

hey there - running into the same issue and I believe it's because withStructuredOutput does not call llm.bind with the tool_choice param that @bracesproul added here: https://github.com/langchain-ai/langchainjs/pull/6195/files.

Also relevant is that it looks like "forced function calling" support with "mode: any" was only recently added for Gemini Flash: https://github.com/google-gemini/generative-ai-js/issues/190. I don't think a change is needed specifically to support flash, but would be great if we could update withStructuredOutput to use forced function calling! Currently withStructuredOutput is unusable at least for me.

I think just accepting a param for tool_choice would work - could have an option that automatically sets tool_choice to the explicitly defined the function name, but since there's only one function setting it to "any" should always work.

[BaharChidem]

Hi there! We are a group of 3 students from the University of Toronto and we are very interested in fixing this issue and also adding some tests. We will submit a PR for this issue by end of November.

[jacoblee93]

Yes please! Will assign it to you @BaharChidem.

[boehlerlukas]

We are running into the same issue. If there is anything we can help with, let me know.

[tonyabracadabra]

I have switched to vercel's generateObject, life is so much easier now!

[austin-duff-prft]

Any update on this issue? Thanks

[bracesproul]

@tonyabracadabra could you send me an example zod schema that will trigger this error?

[tonyabracadabra]

@tonyabracadabra could you send me an example zod schema that will trigger this error?

I believe any zod schema would trigger this if no tools were specified somewhere within

[BaharChidem]

We are continuing to work on the issue and hope to have a PR up by the end of next week. We are currently just finding difficulties with replicating the issue in our Linux VMs, but hope to get this resolved soon. Thanks!

[boehlerlukas]

@BaharChidem thanks a lot!

[BaharChidem]

Hi,

We've been working on implementing forced function calling with tool_choice set to any, focusing on improving the handling of tools within the Google Generative AI integration. The main changes were made in the output_parser and chat_model files. Most of the integration is functioning as expected, and other tests are passing. However, the integration tests specifically related to tool usage, such as bind and invoke tools, streaming tool messages, and invoking tool messages, are failing. We’re unsure if this indicates an issue in our implementation/solution or if the integration tests need to be updated to align with the new functionality. Are we heading in the right direction, or should we revisit the approach? Any advice would be greatly appreciated. Thanks!