griptape icon indicating copy to clipboard operation
griptape copied to clipboard

Create a Groq driver

Open shhlife opened this issue 4 months ago • 0 comments

Is your feature request related to a problem? Please describe. Groq is very fast, and is somewhat OpenAI API compatible.

Using something like this:

import os

from dotenv import load_dotenv
from griptape.drivers import OpenAiChatPromptDriver
from griptape.structures import Agent
from griptape.utils import Chat
from rich import print as print

load_dotenv()
GROQ_API_KEY = os.getenv("GROQ_API_KEY")

agent = Agent(
    prompt_driver=OpenAiChatPromptDriver(
        api_key=GROQ_API_KEY,
        base_url="https://api.groq.com/openai/v1",
        model="llama3-groq-70b-8192-tool-use-preview",
    ),
    stream=True,
)
Chat(agent).start()

is extremely fast - however, if you give it a tool you get this response: WARNING:root:<RetryCallState 2380201637520: attempt #1; slept for 0.0; last result: failed (ValueError Invalid JSON input for ToolAction)

Here's the OpenAI API compatibility doc: https://console.groq.com/docs/openai and tools: https://console.groq.com/docs/tool-use

it says it should work with llama3-groq-70b-8192-tool-use-preview, but it doesn't appear to.

shhlife avatar Oct 04 '24 12:10 shhlife