[Add] Added function calling support to GeminiClient

[arjun-g]

Why are these changes needed?

I have added function calling ability to GeminiClient so that we can use autogen tools with Gemini as well.

Related issue number

Completes "Function Calling" roadmap item in #2387

Checks

• [ ] I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
• [x] I've added tests (if relevant) corresponding to the changes introduced in this PR.
• [ ] I've made sure all auto checks have passed.

[arjun-g]

I am learning python just for the past few days. Kindly give me feedback on what I can improve in this.

[BeibinLi]

Thanks for your great PR @arjun-g !! There are two tiny stuff missing: test cases and tutorial notebook. Can you add test cases for the new features? I can help to update the Gemini notebook after the test cases are done.

[arjun-g]

@BeibinLi added a test case. Let me know if this is suffice

[r4881t]

@arjun-g - A notebook example of using this would be great to have.

[arjun-g]

@BeibinLi added a notebook as well.

[BeibinLi]

Thanks @arjun-g for updating. Take a look at @luxzoli PR #2805 , and we might need to slightly update this PR to be consistent with vertexai package. I will edit the notebook after luxzoli's PR is merged. Then, I will edit this PR and merge it.

[arjun-g]

@BeibinLi got it. I will wait for #2805 to be merged. The function call is a bit different in vertex ai. I will handle that and give the PR.

[arjun-g]

@BeibinLi updated the PR with function calling support for VertexAI as well.

[nileshtrivedi]

Currently unable to use Gemini models in Autogen with Tool use (The LLM responds by itself instead of invoking the tool). Hoping that this PR will fix it.

[arjun-g]

@drewgillson updated the PR as per your comment. Let me know if anything else needs to be done.

[nileshtrivedi]

I wonder if it would be better to have separate implementations for GeminiAI and VertexAI as there are subtle differences in both.

[r4881t]

Hi @arjun-g - Thanks for the amazing work. When I try to use this with a function that has param signature as def some_func(values: str | list[str]) I get an error

/usr/local/lib/python3.10/dist-packages/autogen/oai/gemini.py in oai_function_parameters_to_gemini_function_parameters(function_definition)
    363     The type key is renamed to type_ and the value is capitalized.
    364     """
--> 365     function_definition["type_"] = function_definition["type"].upper()
    366     del function_definition["type"]
    367     if "properties" in function_definition:

KeyError: 'type'

You can find a notebook that demonstrates this error at https://colab.research.google.com/drive/1P56iRkrryHZHz0lA5gZBPur9t7ML_G0A?usp=sharing

Quoting the docs at https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling#python:-array-of-items

Vertex AI offers limited support of the OpenAPI schema. The following attributes are supported: type, nullable, required, format, description, properties, items, enum. The following attributes are not supported: default, optional, maximum, oneOf.

Does this mean that all functions to be used for Gemini should have only 1 type? In this case there should be two functions (a) with param str and another with (b) param list[str] ? If yes, then let's make this clear so devs can update their functions accordingly.

Also note how safety settings have been applied. It would be good to add this as an example in your notebook as well.

[arjun-g]

Hi @arjun-g - Thanks for the amazing work. When I try to use this with a function that has param signature as def some_func(values: str | list[str]) I get an error

/usr/local/lib/python3.10/dist-packages/autogen/oai/gemini.py in oai_function_parameters_to_gemini_function_parameters(function_definition)
    363     The type key is renamed to type_ and the value is capitalized.
    364     """
--> 365     function_definition["type_"] = function_definition["type"].upper()
    366     del function_definition["type"]
    367     if "properties" in function_definition:

KeyError: 'type'

You can find a notebook that demonstrates this error at https://colab.research.google.com/drive/1P56iRkrryHZHz0lA5gZBPur9t7ML_G0A?usp=sharing

Quoting the docs at https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling#python:-array-of-items

Vertex AI offers limited support of the OpenAPI schema. The following attributes are supported: type, nullable, required, format, description, properties, items, enum. The following attributes are not supported: default, optional, maximum, oneOf.

Does this mean that all functions to be used for Gemini should have only 1 type? In this case there should be two functions (a) with param str and another with (b) param list[str] ? If yes, then let's make this clear so devs can update their functions accordingly.

Also note how safety settings have been applied. It would be good to add this as an example in your notebook as well.

Thanks for pointing this out @r4881t . I have added an assertion error for union of data types and deleted the default value.

[r4881t]

Hi @arjun-g , it works now. Thanks.

Another issue I noted is that when i started my app, it complained of importerror and told me to pip install google-generativeai. When I did that and ran again, it again threw the same error. So I added a print statement in the autogen/oai/client.py and found that vertexai module was missing. Then I installed it by pip install google-cloud-aiplatform and it ran successfully.

So a better error message would be helpful

[arjun-g]

Hi @arjun-g , it works now. Thanks.

Another issue I noted is that when i started my app, it complained of importerror and told me to pip install google-generativeai. When I did that and ran again, it again threw the same error. So I added a print statement in the autogen/oai/client.py and found that vertexai module was missing. Then I installed it by pip install google-cloud-aiplatform and it ran successfully.

So a better error message would be helpful

Hi, @r4881t This is unrelated to function calling. This is a problem with vertexai implementation itself. Will give a separate PR for this.

[r4881t]

@BeibinLi @drewgillson

[marklysze]

I'll do some quick tests now...

[marklysze]

Okay, running through a set of test programs (which I've put on my repository), it looks good, however there was an issue with parallel tool calling (when the model suggests two or more tool-calls in one response). It's only handling one tool-call suggestion.

I'll add my code in to correct that.

My repo with test programs that you can use to test: https://github.com/marklysze/AutoGenClientTesting

I ignored the streaming ones as streaming isn't supported yet in the code

[marklysze]

Nice work @arjun-g :)... just need to run pre-commit run to tidy up the test file and notebook, and then commit that. That should rectify the "Code formatting / pre-commit-check" fail that's occurring.

Let us know if you need help with that.

[gitguardian[bot]]

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

Since your pull request originates from a forked repository, GitGuardian is not able to associate the secrets uncovered with secret incidents on your GitGuardian dashboard. Skipping this check run and merging your pull request will create secret incidents on your GitGuardian dashboard.

🔎 Detected hardcoded secret in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
11616921	Triggered	Generic High Entropy Secret	65632487224aae2c7faca67de40fd2bda3ad3905	notebook/agentchat_agentops.ipynb	View secret

🛠 Guidelines to remediate hardcoded secrets

1. Understand the implications of revoking this secret by investigating where it is used in your code.
2. Replace and store your secret safely. Learn here the best practices.
3. Revoke and rotate this secret.
4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

• following these best practices for managing and storing secrets including API keys and other credentials
• install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

---

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 43.75000% with 72 lines in your changes missing coverage. Please review.

Please upload report for BASE (0.2@1582927). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
autogen/oai/gemini.py	43.75%	56 Missing and 16 partials :warning:

Additional details and impacted files

@@          Coverage Diff           @@
##             0.2    #2793   +/-   ##
======================================
  Coverage       ?   25.59%           
======================================
  Files          ?       99           
  Lines          ?    10261           
  Branches       ?     2358           
======================================
  Hits           ?     2626           
  Misses         ?     7325           
  Partials       ?      310           

Flag	Coverage Δ
unittest	12.35% <2.34%> (?)
unittests	24.81% <43.75%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[robraux]

Is there anything else you need here @arjun-g? This is an important component for the gemini client I'd love to make use of in a more official capacity. I'm happy to help wrap up.

[arjun-g]

@robraux I am held up in a work. Will update the PR in couple of days,

[luxzoli]

It would be really nice to have this feature 🙂

[MO7YW4NG]

Looking forward to having this feature 🙂

[luxzoli]

@arjun-g I have prepared a pull request to your repo that would resolve the merge conflicts, synchronize updates from the upstream autogen repo, and also add support for tool config with function calling.

Only changes to gemini.py , test_gemini.py , and openai_utils.py need to be reviewed, as the rest is changed due to synchronization with the official autogen repo except for agentchat_gemini.ipynb, where the difference is only automated formatting, which is not visible to the user anyway and thus irrelevant.

I hope this would save you some effort in case you still do not have time to finish this PR 🙂

Link to the PR: https://github.com/arjun-g/autogen/pull/1

[ekzhu]

@luxzoli would you like to make a PR with your changes directly to this repo? Unless @arjun-g you would like to continue with this?

[luxzoli]

@ekzhu yes, sure. I can make a PR directly to the repo if @arjun-g will not finalize this PR

[ekzhu]

@arjun-g please let us know this week. Thanks!

[ekzhu]

Closing this as work continue in #3726