Function call no working with "gemini-1.5-flash", but it does with "gemini-1.5-pro"

[blopa]

Description of the bug:

Hi. I have a function that consistently return a correct JSON from my function call if I use gemini-1.5-pro, but if I change it to gemini-1.5-flash, I suddenly get no function calls and have all the responses on the text property.

This is my code:

export async function sendChatMessage(messages: any[]) {
    const apiKey = await getApiKey();

    const genAI = new GoogleGenerativeAI(apiKey);

    const conversationContent: Content[] = messages
        .filter((msg) => msg.role !== 'system')
        .map((msg) => ({
            parts: [{ text: msg.content } as Part],
            role: msg.role === 'assistant' ? 'model' : msg.role,
        }));

    const latestMessage = conversationContent.pop();
    const firstMessage = conversationContent.at(0);

    if (firstMessage?.role! !== 'user') {
        conversationContent.unshift({
            parts: [{ text: ' ' } as Part],
            role: 'user',
        });
    }

    const systemParts: Part[] = messages
        .filter((msg) => msg.role === 'system')
        .map((msg) => ({ text: msg.content } as Part));

    const functionDeclarations = [{
        description: "A response to a user's message.",
        name: 'generateMessage',
        parameters: {
            properties: {
                messageToBio: {
                    description: "A message to be saved in the user's bio.",
                    type: 'string',
                },
                messageToUser: {
                    description: [
                        'A message to be displayed to the user.',
                    ].join('\n'),
                    type: 'string',
                },
                shouldGenerateWorkout: {
                    description: 'Whether to generate a workout plan for the user or not.',
                    type: 'boolean',
                },
            },
            required: ['messageToUser', 'shouldGenerateWorkout'],
            type: 'object',
        },
    }];

    const tools: Tool[] = [{ functionDeclarations }];

    const model = genAI.getGenerativeModel({
        generationConfig: {
            maxOutputTokens: 2048,
            temperature: 0.9,
            topK: 1,
            topP: 1
        },
        model: 'gemini-1.5-pro-latest',
        // model: 'gemini-1.5-flash-latest', // doesn't work!
        safetySettings,
        systemInstruction: {
            parts: systemParts,
            role: 'system'
        },
        toolConfig: {
            functionCallingConfig: {
                allowedFunctionNames: functionDeclarations.map((f) => f.name),
                mode: 'ANY',
            },
        } as ToolConfig,
        tools,
    });

    try {
        const chatSession = model.startChat({
            history: conversationContent,
        });

        const result = await chatSession.sendMessage(latestMessage?.parts?.[0].text!);

        if (result.response.promptFeedback && result.response.promptFeedback.blockReason) {
            console.log(`Blocked for ${result.response.promptFeedback.blockReason}`);
            return;
        }

        return {
            messageToBio: '',
            messageToUser: i18n.t('great'),
            shouldGenerateWorkout: false,
            ...result?.response?.candidates?.[0]?.content?.parts?.[0]?.functionCall?.args,
        };
    } catch (e) {
        console.error(e);
        return;
    }
}

@google/generative-ai version: 0.14.0 node.js version: 21.6

Actual vs expected behavior:

Expected:

Get my JSON result inside result?.response?.candidates?.[0]?.content?.parts?.[0]?.functionCall?.args using gemini-1.5-flash or gemini-1.5-pro.

Actual

I get the result inside result?.response?.candidates?.[0]?.content?.parts?.[0]?.text using gemini-1.5-flash.

Any other information you'd like to share?

No response

[navarrodiego]

I think this is intended. Flash model don't support forced function calling with "ANY" parameter.

The tool configuration's ANY mode ("forced function calling") is a Preview feature. It is supported for Gemini 1.5 Pro models only.

https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling

[quan7794]

Try again/ Any type supported by Gemini 1.5 flash

Limitations:

The maximum number of function declarations that can be provided with the request is 128. FunctionCallingConfig.Mode.ANY is available with only the Gemini 1.5 Pro and Gemini 1.5 Flash models

https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/function-calling