cortex.cpp
cortex.cpp copied to clipboard
janhq
[WIP] add support for function calling
This PR adds support for function calling / tool use. Addresses #295
Request:
curl http://localhost:3928/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"messages": [
{
"role": "system",
"content": "You are a helpful assistant with access to some tools. The list of tools and their uses are provided to you. You should respond with the tool to use and appropriate parameters. For example, if there is a calculator tool available and the user asks to do some arithmetic calculation, you can call the calculator() tool.
For eg:
input: What is 42*42?
output: {\"thought\": \"User wants to calculate multiplication of two numbers so I should use the calculator tool\", \"tool\": \"caculator(42*42)\"}
Here is the list of tools provided to you."
},
{
"role": "user",
"content": "What is the weather like in SF?"
},
], "grammar_file": "/path/to/grammar_file",
"tools": "[{
\"type\": \"function\",
\"function\": {
\"name\": \"get_current_weather\",
\"description\": \"Get the current weather in a given location\",
\"parameters\": {
\"type\": \"object\",
\"properties\": {
\"location\": {
\"type\": \"string\",
\"description\": \"The city and state, e.g. San Francisco, CA\"
},
\"unit\": {
\"type\": \"string\",
\"enum\": [\"celsius\", \"fahrenheit\"]
}
},
\"required\": [\"location\"]
}
}
}]",
"tool_choice": "auto"
}'
Response:
{\"thought\": \"User wants to know the weather in San Francisco, so I should use the get_current_weather tool\", \"tool\": \"get_current_weather(location='San Francisco', unit='celsius')\"}
I guess the big question here is what format should the tools input be in if we also want to use llms that are not finetuned for tool use. llama-cpp-python seems to format the tools into something like:
// Supported function definitions that should be called when necessary.
namespace functions {
// Get the current weather in a given location
type get_current_weather = (_: {
// The city and state, e.g. San Francisco, CA
location: string,
// Optional parameter indicating temperature unit
unit?: "celsius" | "fahrenheit",
}) => any;
} // namespace functions
(I could be wrong, this is just from reading the code)
There is an argument to be made that maybe it should resemble a python function instead as most llms should have seen a lot of python code and should know how to invoke python functions better? And then leave it to the consumer to do a validation pass before doing another inference pass. Is there a better way to accomplish this?
In my limited testing with mistral and llama2-chat, I have got decent results with just passing the json as it is unless the user intentionally sends malformed input.
@hiro-v It actually works with the current code changes, but I just wanted to open a discussion in case someone had any suggestions on how to format the function signature before merging it.
It's pretty outdated, any update on this @tohrnii? Could you please share an example or so and reopen after rebased?
