Adding support for OpenAI functions

[nc]

WIP. Adding support for OpenAI functions. Would love feedback.

[nc]

@microsoft-github-policy-service agree company="Teamspace Technologies Ltd."

[slundberg]

Thanks! Was also looking into this and will review this shortly. One goal here is to support functions in a general way that encourages consistency with both commercial and open-source models

[bhy]

Since one guidance program can send multiple API request with multiple gen commands, each can have different set of functions , so ideally we’d like to have the functions as arguments to gen instead of the whole program.

[slundberg]

@bhy There are two parts to function definition, one is defining the function for the model to use, and the other is when you tell the model a subset of those functions it is allowed to use right now. The former will often be program wide I think, while the latter will be specific to each gen command.

[slundberg]

Just took a quick look. I like the new function role, I think for the function definitions we will need to tweak things a bit to allow for this to work for open models as well as the OpenAI API.

I would also like to support just passing a documented python function directly :)

[slundberg]

I am thinking about an interface that looks like this (roughly). It runs a loop of function calls until you get an answer. Thoughts?

import guidance
import json

def get_current_weather(location, unit="fahrenheit"):
    """ Get the current weather in a given location.
    
    Parameters
    ----------
    location : string
        The location to get the weather for.
    unit : str, optional
        The unit to get the weather in. Can be either "fahrenheit" or "celsius".
    """
    weather_info = {
        "location": location,
        "temperature": "72",
        "unit": unit,
        "forecast": ["sunny", "windy"],
    }
    return json.dumps(weather_info)

program = guidance("""
{{~#system~}}
You are a helpful assistant.
{{~function_def get_current_weather}}
{{~/system~}}

{{~#user~}}
Get the current weather in New York City.
{{~/user~}}

{{~#while True~}}
    {{~#assistant~}}
    {{gen 'answer' temperature=1.0 max_tokens=50 functions="auto"}}
    {{~/assistant~}}

    {{#if answer_function_call}}
        {{~#function~}}
        {{answer_function_call()}}
        {{~/function~}}
    {{else}}
        {{break}}
    {{/if}}
{{/while}}
""")

executed_program = program(get_current_weather=get_current_weather)
executed_program["answer"]

[nc]

@slundberg that looks great! I'm not sure how much time i'm going to have to get this over the line in the next day or two. If you want to pick it up please go ahead :). Otherwise I'll get to it by the end of the weekend.

Btw I think a good litmus test would be if you can implement chain of thought prompting with a set of tools using that interface. This is what we use at the moment without OpenAI functions support:

{{#system~}}
{{agent_description}}

{{#each tools}}
{{~this.name}}: {{this.description}}
{{~/each}}
Use the following format:

Question: the input question you must answer
Thought: you should always think about what to do and then do it
Action: the action to take, should be one of [{{~#each tools}}{{this.name}}{{#unless @last}}, {{/unless}}{{~/each}}]
Action Input: the input to the action
Observation: the result of the action
... (this Thought/Action/Action Input/Observation can repeat N times)
Thought: I now know the final answer
Final Answer: the final answer to the original input question

Begin!

Question: {{question}}
Thought: {{initial_thought}}
{{~/system}}
{{~#geneach 'loop' stop=False}}
{{#assistant~}}
{{gen 'this.thought' temperature=0 max_tokens=1500 stop_regex='Observation:'}}
{{~/assistant}}
{{~#if (contains this.thought 'Final Answer:')}}{{break}}{{/if~}}
{{#assistant~}}
Observation: {{await 'observation'}}
{{~/assistant}}
{{~/geneach}}

[nc]

One gotcha I found is is openai.ChatCompletion.acreate fails with functions provided. openai.ChatCompletion.create works though. https://github.com/openai/openai-python/issues/488 more info here.

[slundberg]

Great! I'll work on it and see where I get. This is my latest refinement:

{{~#system~}}
You are a helpful assistant.
{{~function_def get_current_weather}}
{{~/system~}}

{{~#user~}}
Get the current weather in New York City.
{{~/user~}}

{{~#while True~}}
    {{~#assistant~}}
    {{gen 'answer' temperature=1.0 max_tokens=50 functions="auto"}}
    {{~/assistant~}}

    {{#if callable(answer)}}
        {{~#function~}}
        {{answer()}}
        {{~/function~}}
    {{else}}
        {{break}}
    {{/if}}
{{~/while~}}

The answer just becomes a callable object when a function call happens (and you can still get whatever content was also generated by calling str(answer))

[andaag]

Some flag for maximum number of loops would be good to prevent runaways, but other than that looks very nice 👍

[slundberg]

Need to merge and won't likely be able to do that until Monday. But here is what I have in case any of you have comments:

I'll describe the added features to make this work later.

program = guidance("""
{{~#system~}}
You are a helpful assistant.
{{>@tool_def}}
{{~/system~}}

{{~#user~}}
Get the current weather in New York City.
{{~/user~}}

{{~#each range(100)~}}
    {{~#assistant~}}
    {{gen 'answer' temperature=1.0 max_tokens=50 function_call="auto"}}
    {{~/assistant~}}

    {{#if callable(answer)}}
        {{~#function name=answer.function_name~}}
        {{answer()}}
        {{~/function~}}
    {{else}}
        {{break}}
    {{/if}}
{{~/each~}}""")

out = program(functions=[
    {
        "name": "get_current_weather",
        "description": "Get the current weather in a given location",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {
                    "type": "string",
                    "description": "The city and state, e.g. San Francisco, CA",
                },
                "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
            },
            "required": ["location"],
        }
    }
], get_current_weather=get_current_weather)

which gives the output

[nc]

Looks great! Is tooldef required? And does it always add to the system message? Or is it for functions support for other LLMs? Suspect adding additional info to the system prompt could have desirable/undesirable effects.

[slundberg]

The way the GPT models work on the backend (according to the docs) is that they insert function definitions into the system message. This interface this just exposes that directly so we write the system messages just like they will be given to the model, and then the OpenAI LLM object just converts those into API calls. So @tool_def is a guidance program specified by the LLM to dump function definitions. For other LLMs we may want to put them anywhere, but for OpenAI models, this is where they should go since that is what the model were fine tuned on.

[oggreo]

This looks great! Was there any reference in the docs where they specified the format for the tool_def in the system call? I did find where it says the function definition is injected into the system call but couldn't find the format of how it should be.

[bhy]

@slundberg Just being curious about thistypescript code in the assistant message:

functions.get_current_weather({
"location": "New York City"
})

Is this what ChatGPT actually generates behind the scenes? Since for returning function calls, we are supposed to send back this assistant message along with function call results in a new function message, is that OK to send this as typescript code instead of the function_call JSON object?

[slundberg]

@slundberg Just being curious about thistypescript code in the assistant message:

functions.get_current_weather({
"location": "New York City"
})

Is this what ChatGPT actually generates behind the scenes? Since for returning function calls, we are supposed to send back this assistant message along with function call results in a new function message, is that OK to send this as typescript code instead of the function_call JSON object?

Note that the portion you quoted is sent by the model back to us. It is the system type refs that get sent to the model. I don't know if you can manually insert those or not (you might be able to), but in Guidance we just detect when you have a system message in this format and convert it to an API call arg.

I don't know for sure what format OpenAI uses. I expect there is probably some special token involved, but this format is what ChatGPT says it uses. That is how I picked all the formats here, I just asked ChatGPT what format it expects functions to be in and how it calls functions.

[nc]

Oh! In that case we should make this customisable, i suspect there is a gain in performance to be made if Guidance doesn’t assume a format but takes a provided one.

Sent via Superhuman iOS @.***>

On Mon, Jun 19 2023 at 8:04 pm, Scott Lundberg @.***> wrote:

@slundberg https://github.com/slundberg Just being curious about this typescript code in the assistant message:

functions.get_current_weather({"location": "New York City"})

Is this what ChatGPT actually generates behind the scenes? Since for returning function calls, we are supposed to send back this assistant message along with function call results in a new function message, is that OK to send this as typescript code instead of the function_call JSON object?

Note that the portion you quoted is sent by the model back to us. It is the system type refs that get sent to the model. I don't know if you can manually insert those or not (you might be able to), but in Guidance we just detect when you have a system message in this format and convert it to an API call arg.

I don't know for sure what format OpenAI uses. I expect there is probably some special token involved, but this format is what ChatGPT says it uses. That is how I picked all the formats here, I just asked ChatGPT what format it expects functions to be in and how it calls functions.

— Reply to this email directly, view it on GitHub https://github.com/microsoft/guidance/pull/239#issuecomment-1597630078, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAGCP5T4XPWTJKW22VM6X3XMCPDZANCNFSM6AAAAAAZIEKB2U . You are receiving this because you were mentioned.Message ID: @.***>

[codecov-commenter]

Codecov Report

Merging #239 (79ebd4b) into main (5f7fa7f) will decrease coverage by 1.52%. The diff coverage is 53.64%.

@@            Coverage Diff             @@
##             main     #239      +/-   ##
==========================================
- Coverage   70.15%   68.63%   -1.52%     
==========================================
  Files          49       54       +5     
  Lines        2858     3042     +184     
==========================================
+ Hits         2005     2088      +83     
- Misses        853      954     +101     

Impacted Files	Coverage Δ
guidance/library/_geneach.py	77.27% <0.00%> (ø)
guidance/_program.py	57.22% <23.80%> (-0.65%)	:arrow_down:
guidance/llms/_openai.py	27.57% <27.05%> (+0.38%)	:arrow_up:
guidance/library/_callable.py	28.57% <28.57%> (ø)
guidance/library/_not.py	50.00% <50.00%> (ø)
guidance/llms/_llm.py	78.78% <50.00%> (-6.03%)	:arrow_down:
guidance/_program_executor.py	80.62% <55.17%> (-2.53%)	:arrow_down:
guidance/library/_each.py	79.59% <61.90%> (-14.16%)	:arrow_down:
guidance/library/_len.py	66.66% <66.66%> (ø)
guidance/library/_range.py	66.66% <66.66%> (ø)
... and 14 more

... and 1 file with indirect coverage changes

[cschubiner]

excited for this to come out!

[slundberg]

I have started a documentation / tutorial notebook here: https://github.com/microsoft/guidance/blob/561108f4f15ba930f26f1ff9c49afd0b7a543c39/notebooks/art_of_prompt_design/tool_use.ipynb FYI

[bhy]

I don't know for sure what format OpenAI uses. I expect there is probably some special token involved, but this format is what ChatGPT says it uses. That is how I picked all the formats here, I just asked ChatGPT what format it expects functions to be in and how it calls functions.

If you look at the OpenAI guide, the example code is sending back the assistant message containing the function_call JSON object: (line 62)

        messages.append(response_message)  # extend conversation with assistant's reply

So is reformatting the function_call JSON object as typescript code equivalent to this?

[cschubiner]

Ready to merge?

[JoseMoreville]

Does this work with the new role functions?