Functions support with local LLMs

[einherjer0]

I see similar issue #672, but it's old and the answer doesn't seem to address functions specifically, just general installation, so I wanted to try again. I also see issue #612 where someone seems to have figured it out, but run into another issue.

I'm hosting ollama, integrated it with shell_gpt, and I would like to try out the functions feature. shell_gpt seems to be aware of my functions and picks one correctly, but doesn't execute it. Instead, the following is returned:

$ sgpt "what files are in $(pwd)"
{"name": "execute_shell_command", "arguments": {"shell_command": "ls -la /home/user1"}} 

It seems to have found the default execute_shell.py, but doesn't run the "execute()" function.

$ sgpt "open google.com"
{"name": "open_url_in_browser", "arguments": {"url": "https://google.com"}} 

I added open_url_in_browser.py file, which it also found and tried to use correctly. Again, didn't actually execute. With some debugging, I proved that Config() class is being instantiated, but execute() doesn't get called.

Is there a way to fix this?

Here is my .sgptrc configuration:

CHAT_CACHE_PATH=/tmp/chat_cache
CACHE_PATH=/tmp/cache
CHAT_CACHE_LENGTH=100
CACHE_LENGTH=100
REQUEST_TIMEOUT=60
DEFAULT_MODEL=ollama/mistral:7b-instruct
DEFAULT_COLOR=magenta
ROLE_STORAGE_PATH=/home/user1/.config/shell_gpt/roles
DEFAULT_EXECUTE_SHELL_CMD=true
DISABLE_STREAMING=false
CODE_THEME=dracula
OPENAI_FUNCTIONS_PATH=/home/user1/.config/shell_gpt/functions
OPENAI_USE_FUNCTIONS=true
SHOW_FUNCTIONS_OUTPUT=true
API_BASE_URL=http://localhost:11434
PRETTIFY_MARKDOWN=true
USE_LITELLM=true
SHELL_INTERACTION=true
OS_NAME=auto
SHELL_NAME=auto
OPENAI_API_KEY=false

[TheR1D]

ShellGPT currently supports function calling only with OpenAI API.

[lemassykoi]

Ollama is Open-AI compatible:

https://ollama.com/blog/openai-compatibility

[maxregan]

Indeed it seems largely compatible but not exactly. I can confirm that I am able to run functions using local models, but I had to remove a parameter in handler.py

        if functions:
            additional_kwargs["tool_choice"] = "auto"
            additional_kwargs["tools"] = functions
            #additional_kwargs["parallel_tool_calls"] = False # litellm complains

However, every model I tried hallucinated non-existent functions. I only had the included shell function, and when I prompted with something that didn't clearly need a shell function, they tried making up a function to call (e.g. greet_user).

[lemassykoi]

However, every model I tried hallucinated non-existent functions. I only had the included shell function, and when I prompted with something that didn't clearly need a shell function, they tried making up a function to call (e.g. greet_user).

Try with a stronger model. From Ollama Git Issues, I learned that to have perfect tool calls, I need a strong model.

https://ollama.com/library/qwq always succeed

light but still good results: qwen2.5:14b-instruct