add a hacky form of google search grounding for vertexai models

[asleepynerd]

Add a hacky form of grounding for google search until I investigate how it works better and update it.

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
litellm	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Apr 25, 2024 7:10pm

[krrishdholakia]

hey - can someone point me to docs on google search grounding for vertex ai?

[asleepynerd]

hey - can someone point me to docs on google search grounding for vertex ai?

https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/grounding?hl=en

[krrishdholakia]

@joshiewtf can we make this simpler for a user

grounding is a tool, so perhaps it can just be a string in the list of tools that we can handle specially?

from openai import OpenAI
client = OpenAI()

tools = ["grounding"]
messages = [{"role": "user", "content": "What's the weather like in Boston today?"}]
completion = client.chat.completions.create(
  model="gemini-1.5-pro",
  messages=messages,
  tools=tools,
  tool_choice="auto"
)

print(completion)

this looks similar to our callback flow

what do you think? @joshiewtf @Manouchehri

[asleepynerd]

@krrishdholakia from what I’ve seen, you would need to redefine how the tools are parsed on the proxy/package. because grounding is a tool in itself, yes but it doesn’t follow any openai/openapi spec at all. Theres also other parameters, e.g you can have a google search grounding, which is what I’ve implemented, but then you can have vertex dataset grounding, which reads info from a dataset stored on gcp.

[krrishdholakia]

Closing as this is now live - https://docs.litellm.ai/docs/providers/vertex#grounding