Is Function Calling possible?

[jaydanurwin]

I've looked into the samples a bit and the available params but I'm not seeing anything about function calling/tools. Is this already possible and I'm missing it or is it planned to be supported? It would be awesome to be able to call local Dart functions through Gemini similar to how it appears to be possible with Python or a REST API

[MrCsabaToth]

Badly needed

[davidmigloz]

It's already supported. https://ai.google.dev/gemini-api/docs/function-calling/tutorial?lang=dart

[MrCsabaToth]

That'd be dope! I'll try that next week!

[adamglin0]

It's already supported. https://ai.google.dev/gemini-api/docs/function-calling/tutorial?lang=dart

I try function call by this post, but is not works. Gemini return this:

I need to access a live exchange rate API to answer your question. I do not have access to real-time information, including currency exchange rates.  Therefore, I cannot provide you with the current exchange rate for US dollars to Swedish krona. 

To get the most up-to-date information, I recommend checking a reliable online converter or your bank's website. 

[MrCsabaToth]

I try function call by this post, but is not works. Gemini return this:

I need to access a live exchange rate API to answer your question. I do not have access to real-time information, including currency exchange rates.  Therefore, I cannot provide you with the current exchange rate for US dollars to Swedish krona. 

To get the most up-to-date information, I recommend checking a reliable online converter or your bank's website. 

What's the code you are configuring the function / tool with, and then calling Gemini?

[adamglin0]

@MrCsabaToth

Sorry, if I set model as 'gemini-1.5-flash' it works!

[MrCsabaToth]

@MrCsabaToth

Sorry, if I set model as 'gemini-1.5-flash' it works!

No problem but I'm still highly interested in the source code

[MrCsabaToth]

Actually now I'm at the point that I'm trying function calling and it doesn't seem to be working. I was providing a function to retrieve weather data. I tried both gemini-1.5-flash-latest and gemini-1.5-pro-latest with the prompt "What is the weather today?", and I got back I am sorry, I cannot provide you with the weather information. I do not have access to real-time data like weather conditions.. Note that I also provide a tool to retrieve the location.

[adamglin0]

Actually now I'm at the point that I'm trying function calling and it doesn't seem to be working. I was providing a function to retrieve weather data. I tried both gemini-flash-lates and gemini-pro-latest with the prompt "What is the weather today?", and I got back I am sorry, I cannot provide you with the weather information. I do not have access to real-time data like weather conditions.. Note that I also provide a tool to retrieve the location.

Do you add function_calling_config parameter? Try to delete this.

[MrCsabaToth]

Actually now I'm at the point that I'm trying function calling and it doesn't seem to be working. I was providing a function to retrieve weather data. I tried both gemini-flash-lates and gemini-pro-latest with the prompt "What is the weather today?", and I got back I am sorry, I cannot provide you with the weather information. I do not have access to real-time data like weather conditions.. Note that I also provide a tool to retrieve the location.

Do you add function_calling_config parameter? Try to delete this.

No, do you have some code by any chance? function_calling_config seems to be Pythonic, we are in Dart land here. However the Dart API also has a FunctionCallingMode. It has modes: auto, any, and none. any forces a function call, auto supposed to be the default. I'll try to add a toolConfig: ToolConfig(functionCallingConfig: FunctionCallingConfig(mode: FunctionCallingMode.auto)) during model instantiation. Unlike the Python and some other language APIs the chat instantiation doesn't have a parameter like start_chat(enable_automatic_function_calling=True) in Python. The Dart startChat only has history, safetySettings, and generationCOnfig. This latter doesn't have function call related settings in it.

@adamglin0 How did you solve the problem?

[MrCsabaToth]

I tested explicitly specifying toolConfig: ToolConfig(functionCallingConfig: FunctionCallingConfig(mode: FunctionCallingMode.auto)) and it didn't help. My source code: https://github.com/CsabaConsulting/InspectorGadgetApp/blob/21cfcc45436dd1c155c703183d22c0239376ee56/lib/interaction/view/interaction_page.dart#L257

[adamglin0]

Actually now I'm at the point that I'm trying function calling and it doesn't seem to be working. I was providing a function to retrieve weather data. I tried both gemini-flash-lates and gemini-pro-latest with the prompt "What is the weather today?", and I got back I am sorry, I cannot provide you with the weather information. I do not have access to real-time data like weather conditions.. Note that I also provide a tool to retrieve the location.

Do you add function_calling_config parameter? Try to delete this.

No, do you have some code by any chance? function_calling_config seems to be Pythonic, we are in Dart land here. However the Dart API also has a FunctionCallingMode. It has modes: auto, any, and none. any forces a function call, auto supposed to be the default. I'll try to add a toolConfig: ToolConfig(functionCallingConfig: FunctionCallingConfig(mode: FunctionCallingMode.auto)) during model instantiation. Unlike the Python and some other language APIs the chat instantiation doesn't have a parameter like start_chat(enable_automatic_function_calling=True) in Python. The Dart startChat only has history, safetySettings, and generationCOnfig. This latter doesn't have function call related settings in it.

@adamglin0 How did you solve the problem?

Let me find my old code, this will take some time.

[adamglin0]

@MrCsabaToth I can no longer reproduce the issue. But I can tell you how I resolved it.

1. After I found the same error, I tried this document.

But still not successful.

2. Then I found that my GenerativeModel section seemed different from the one in the document, so I copied part of the model and it worked. https://ai.google.dev/gemini-api/docs/function-calling/tutorial?hl=zh-cn&lang=dart

3. Later, after leaving a message on this issue, I modified the value of the model, but then realized that I wrote toolConfig in GenerativeModel (it seems like this because there is no git record).

4. Now I cannot reproduce it anymore, sorry for that

[MrCsabaToth]

Reporting back about what I've found out: @adamglin0 was exactly right here https://github.com/google-gemini/generative-ai-dart/issues/81#issuecomment-2172053459

If I go with gemini-1.5-flash or gemini-1.5-pro instead of gemini-1.5-flash-latest or gemini-1.5-pro-latest (so not using the -latest model name prefix) the tool functionality seems to be waking up. Now I'm getting I need to know your location to get the weather forecast. Can you tell me your latitude and longitude?. So it is probably seeing my weather function, however it misses the location function which would exactly provide the lat / long.

Since https://github.com/google-gemini/generative-ai-dart/blob/main/samples/dart/bin/function_calling.dart demonstrates a function call chain possibility, I expect the model to figure out the two step of first calling the location function for the lat long GPS, and then use those as parameters to call the weather function.

[MrCsabaToth]

I'm dealing with a problem with multiple tools: #194