magentic icon indicating copy to clipboard operation
magentic copied to clipboard
verified publisher icon jackmpcollins

How to do both text output and FunctionCall in the same request?

Open jaromiru opened this issue 1 year ago • 3 comments

Hello, I want the model to reason before making a function call in one request, which is possible in other libraries. However, I haven't found a way to do this in magentic. The usual signature of a method is:

def do_something(param: str) -> FunctionCall: ...

which does not allow any other outputs than the function call.

I'd be happy to do something like:

def do_something(param: str) -> tuple[Annotated[str, "reasoning"], FunctionCall]: ...

but this results in a pydantic error. It applies to ParallelFunctionCall also.

jaromiru avatar Jun 06 '24 13:06 jaromiru

Hi @jaromiru

The OpenAI API returns either a message or tool call(s), while the Anthropic API can return "thoughts" before making a tool call (see related issue https://github.com/jackmpcollins/magentic/issues/220). Because of this, a return type that allows "thinking" before making a function call would only work with some LLM providers, but it might still be useful to have.

One approach you could try is having separate steps for the thinking and the function call. Something like

@prompt(...)
def think_about_doing_something(param: str) -> str: ...

@prompt("Based on thoughts {thoughts}. Do thing")
def do_something(thoughts: str, param: str) -> str: ...

thoughts = think_about_doing_something(param)
function_call = do_something(thoughts, param)

Another option which would reduce this to a single query would be to set the return type to a pydantic model with an "explanation" field and fields for the function parameters. You would need to do this for all available functions and union them in the return type. Similar to https://magentic.dev/structured-outputs/#chain-of-thought-prompting

class ExplainedFunctionParams(BaseModel):
    explanation: str = Field(description="reasoning for the following choice of parameters")
    param: str

@prompt(...)  # No `functions` provided here
def do_something(param: str) -> ExplainedFunctionParams: ...

func_params = do_something(param)
my_function(func_params.param)

Please let me know if either of these approaches would work for you. Thanks for using magentic.

jackmpcollins avatar Jun 17 '24 03:06 jackmpcollins

Hi @jackmpcollins, thanks for the answer and the proposed workarounds. Obviously, both of them have their drawbacks. My main motivation was to reduce costs by having one LLM call (especially when the context is large), and the first method does not help with this. The second proposal makes it complicated, brittle, hand-engineered, prone to errors and avoids the ease-of-use of Magentic library.

I currently have no solutions on my own, so we can close the ticket, or keep it open for followup discussion.

jaromiru avatar Jun 27 '24 11:06 jaromiru

@jaromiru Let's leave this open because it would be great for magentic to support this for the Anthropic API. Please let me know if/how you solve this for OpenAI using another library or their API directly because that could inform how to solve it generally for magentic.

jackmpcollins avatar Jun 27 '24 20:06 jackmpcollins

@jackmpcollins Maybe it's finally a good time to revive this given the new release of Sonnet 3.5!

It doesn't look like Anthropic will be changing this pattern anytime soon.

mnicstruwig avatar Oct 23 '24 08:10 mnicstruwig

GPT-4o also supports thinking before tool use. I'll try to get this added soon. Probably will be a new return type that is an iterable of StreamedStr and FunctionCall, similar to how ParallelFunctionCall is an iterable of FunctionCall.

from openai import Client

client = Client()

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Say hello, then call the tool for Boston"}],
    stream=True,
    stream_options={"include_usage": True},
    tools=[
        {
            "type": "function",
            "function": {
                "name": "get_current_weather",
                "description": "Get the current weather in a given location",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "location": {
                            "type": "string",
                            "description": "The city and state, e.g. San Francisco, CA",
                        },
                        "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                    },
                    "required": ["location"],
                },
            },
        },
    ],
)
for chunk in response:
    print(chunk)

ChatCompletionChunk(id='chatcmpl-AN7qZSMv8i5J9Mrm6IgFgM0lrl9wh', choices=[Choice(delta=ChoiceDelta(content='', function_call=None, refusal=None, role='assistant', tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1730075071, model='gpt-4o-2024-08-06', object='chat.completion.chunk', service_tier=None, system_fingerprint='fp_72bbfa6014', usage=None)
ChatCompletionChunk(id='chatcmpl-AN7qZSMv8i5J9Mrm6IgFgM0lrl9wh', choices=[Choice(delta=ChoiceDelta(content='Hello', function_call=None, refusal=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1730075071, model='gpt-4o-2024-08-06', object='chat.completion.chunk', service_tier=None, system_fingerprint='fp_72bbfa6014', usage=None)
ChatCompletionChunk(id='chatcmpl-AN7qZSMv8i5J9Mrm6IgFgM0lrl9wh', choices=[Choice(delta=ChoiceDelta(content=' there', function_call=None, refusal=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1730075071, model='gpt-4o-2024-08-06', object='chat.completion.chunk', service_tier=None, system_fingerprint='fp_72bbfa6014', usage=None)
ChatCompletionChunk(id='chatcmpl-AN7qZSMv8i5J9Mrm6IgFgM0lrl9wh', choices=[Choice(delta=ChoiceDelta(content='!', function_call=None, refusal=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1730075071, model='gpt-4o-2024-08-06', object='chat.completion.chunk', service_tier=None, system_fingerprint='fp_72bbfa6014', usage=None)
ChatCompletionChunk(id='chatcmpl-AN7qZSMv8i5J9Mrm6IgFgM0lrl9wh', choices=[Choice(delta=ChoiceDelta(content=' Let', function_call=None, refusal=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1730075071, model='gpt-4o-2024-08-06', object='chat.completion.chunk', service_tier=None, system_fingerprint='fp_72bbfa6014', usage=None)
ChatCompletionChunk(id='chatcmpl-AN7qZSMv8i5J9Mrm6IgFgM0lrl9wh', choices=[Choice(delta=ChoiceDelta(content=' me', function_call=None, refusal=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1730075071, model='gpt-4o-2024-08-06', object='chat.completion.chunk', service_tier=None, system_fingerprint='fp_72bbfa6014', usage=None)
ChatCompletionChunk(id='chatcmpl-AN7qZSMv8i5J9Mrm6IgFgM0lrl9wh', choices=[Choice(delta=ChoiceDelta(content=' get', function_call=None, refusal=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1730075071, model='gpt-4o-2024-08-06', object='chat.completion.chunk', service_tier=None, system_fingerprint='fp_72bbfa6014', usage=None)
ChatCompletionChunk(id='chatcmpl-AN7qZSMv8i5J9Mrm6IgFgM0lrl9wh', choices=[Choice(delta=ChoiceDelta(content=' the', function_call=None, refusal=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1730075071, model='gpt-4o-2024-08-06', object='chat.completion.chunk', service_tier=None, system_fingerprint='fp_72bbfa6014', usage=None)
ChatCompletionChunk(id='chatcmpl-AN7qZSMv8i5J9Mrm6IgFgM0lrl9wh', choices=[Choice(delta=ChoiceDelta(content=' current', function_call=None, refusal=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1730075071, model='gpt-4o-2024-08-06', object='chat.completion.chunk', service_tier=None, system_fingerprint='fp_72bbfa6014', usage=None)
ChatCompletionChunk(id='chatcmpl-AN7qZSMv8i5J9Mrm6IgFgM0lrl9wh', choices=[Choice(delta=ChoiceDelta(content=' weather', function_call=None, refusal=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1730075071, model='gpt-4o-2024-08-06', object='chat.completion.chunk', service_tier=None, system_fingerprint='fp_72bbfa6014', usage=None)
ChatCompletionChunk(id='chatcmpl-AN7qZSMv8i5J9Mrm6IgFgM0lrl9wh', choices=[Choice(delta=ChoiceDelta(content=' for', function_call=None, refusal=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1730075071, model='gpt-4o-2024-08-06', object='chat.completion.chunk', service_tier=None, system_fingerprint='fp_72bbfa6014', usage=None)
ChatCompletionChunk(id='chatcmpl-AN7qZSMv8i5J9Mrm6IgFgM0lrl9wh', choices=[Choice(delta=ChoiceDelta(content=' Boston', function_call=None, refusal=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1730075071, model='gpt-4o-2024-08-06', object='chat.completion.chunk', service_tier=None, system_fingerprint='fp_72bbfa6014', usage=None)
ChatCompletionChunk(id='chatcmpl-AN7qZSMv8i5J9Mrm6IgFgM0lrl9wh', choices=[Choice(delta=ChoiceDelta(content=' for', function_call=None, refusal=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1730075071, model='gpt-4o-2024-08-06', object='chat.completion.chunk', service_tier=None, system_fingerprint='fp_72bbfa6014', usage=None)
ChatCompletionChunk(id='chatcmpl-AN7qZSMv8i5J9Mrm6IgFgM0lrl9wh', choices=[Choice(delta=ChoiceDelta(content=' you', function_call=None, refusal=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1730075071, model='gpt-4o-2024-08-06', object='chat.completion.chunk', service_tier=None, system_fingerprint='fp_72bbfa6014', usage=None)
ChatCompletionChunk(id='chatcmpl-AN7qZSMv8i5J9Mrm6IgFgM0lrl9wh', choices=[Choice(delta=ChoiceDelta(content='.', function_call=None, refusal=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1730075071, model='gpt-4o-2024-08-06', object='chat.completion.chunk', service_tier=None, system_fingerprint='fp_72bbfa6014', usage=None)
ChatCompletionChunk(id='chatcmpl-AN7qZSMv8i5J9Mrm6IgFgM0lrl9wh', choices=[Choice(delta=ChoiceDelta(content=None, function_call=None, refusal=None, role=None, tool_calls=[ChoiceDeltaToolCall(index=0, id='call_rJqb5hD1Yhot4BFzRm8AbIis', function=ChoiceDeltaToolCallFunction(arguments='', name='get_current_weather'), type='function')]), finish_reason=None, index=0, logprobs=None)], created=1730075071, model='gpt-4o-2024-08-06', object='chat.completion.chunk', service_tier=None, system_fingerprint='fp_72bbfa6014', usage=None)
ChatCompletionChunk(id='chatcmpl-AN7qZSMv8i5J9Mrm6IgFgM0lrl9wh', choices=[Choice(delta=ChoiceDelta(content=None, function_call=None, refusal=None, role=None, tool_calls=[ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='{"', name=None), type=None)]), finish_reason=None, index=0, logprobs=None)], created=1730075071, model='gpt-4o-2024-08-06', object='chat.completion.chunk', service_tier=None, system_fingerprint='fp_72bbfa6014', usage=None)
ChatCompletionChunk(id='chatcmpl-AN7qZSMv8i5J9Mrm6IgFgM0lrl9wh', choices=[Choice(delta=ChoiceDelta(content=None, function_call=None, refusal=None, role=None, tool_calls=[ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='location', name=None), type=None)]), finish_reason=None, index=0, logprobs=None)], created=1730075071, model='gpt-4o-2024-08-06', object='chat.completion.chunk', service_tier=None, system_fingerprint='fp_72bbfa6014', usage=None)
ChatCompletionChunk(id='chatcmpl-AN7qZSMv8i5J9Mrm6IgFgM0lrl9wh', choices=[Choice(delta=ChoiceDelta(content=None, function_call=None, refusal=None, role=None, tool_calls=[ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='":"', name=None), type=None)]), finish_reason=None, index=0, logprobs=None)], created=1730075071, model='gpt-4o-2024-08-06', object='chat.completion.chunk', service_tier=None, system_fingerprint='fp_72bbfa6014', usage=None)
ChatCompletionChunk(id='chatcmpl-AN7qZSMv8i5J9Mrm6IgFgM0lrl9wh', choices=[Choice(delta=ChoiceDelta(content=None, function_call=None, refusal=None, role=None, tool_calls=[ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='Boston', name=None), type=None)]), finish_reason=None, index=0, logprobs=None)], created=1730075071, model='gpt-4o-2024-08-06', object='chat.completion.chunk', service_tier=None, system_fingerprint='fp_72bbfa6014', usage=None)
ChatCompletionChunk(id='chatcmpl-AN7qZSMv8i5J9Mrm6IgFgM0lrl9wh', choices=[Choice(delta=ChoiceDelta(content=None, function_call=None, refusal=None, role=None, tool_calls=[ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments=',', name=None), type=None)]), finish_reason=None, index=0, logprobs=None)], created=1730075071, model='gpt-4o-2024-08-06', object='chat.completion.chunk', service_tier=None, system_fingerprint='fp_72bbfa6014', usage=None)
ChatCompletionChunk(id='chatcmpl-AN7qZSMv8i5J9Mrm6IgFgM0lrl9wh', choices=[Choice(delta=ChoiceDelta(content=None, function_call=None, refusal=None, role=None, tool_calls=[ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments=' MA', name=None), type=None)]), finish_reason=None, index=0, logprobs=None)], created=1730075071, model='gpt-4o-2024-08-06', object='chat.completion.chunk', service_tier=None, system_fingerprint='fp_72bbfa6014', usage=None)
ChatCompletionChunk(id='chatcmpl-AN7qZSMv8i5J9Mrm6IgFgM0lrl9wh', choices=[Choice(delta=ChoiceDelta(content=None, function_call=None, refusal=None, role=None, tool_calls=[ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='"}', name=None), type=None)]), finish_reason=None, index=0, logprobs=None)], created=1730075071, model='gpt-4o-2024-08-06', object='chat.completion.chunk', service_tier=None, system_fingerprint='fp_72bbfa6014', usage=None)
ChatCompletionChunk(id='chatcmpl-AN7qZSMv8i5J9Mrm6IgFgM0lrl9wh', choices=[Choice(delta=ChoiceDelta(content=None, function_call=None, refusal=None, role=None, tool_calls=None), finish_reason='tool_calls', index=0, logprobs=None)], created=1730075071, model='gpt-4o-2024-08-06', object='chat.completion.chunk', service_tier=None, system_fingerprint='fp_72bbfa6014', usage=None)
ChatCompletionChunk(id='chatcmpl-AN7qZSMv8i5J9Mrm6IgFgM0lrl9wh', choices=[], created=1730075071, model='gpt-4o-2024-08-06', object='chat.completion.chunk', service_tier=None, system_fingerprint='fp_72bbfa6014', usage=CompletionUsage(completion_tokens=32, prompt_tokens=81, total_tokens=113, completion_tokens_details=CompletionTokensDetails(audio_tokens=None, reasoning_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=None, cached_tokens=0)))

jackmpcollins avatar Oct 28 '24 00:10 jackmpcollins

@jaromiru @mnicstruwig

This should now be resolved by StreamedResponse in https://github.com/jackmpcollins/magentic/releases/tag/v0.34.0 Please let me know if this works for you! Thanks

Example from there

from magentic import prompt, FunctionCall, StreamedResponse, StreamedStr


def get_weather(city: str) -> str:
    return f"The weather in {city} is 20°C."


@prompt(
    "Say hello, then get the weather for: {cities}",
    functions=[get_weather],
)
def describe_weather(cities: list[str]) -> StreamedResponse: ...


response = describe_weather(["Cape Town", "San Francisco"])
for item in response:
    if isinstance(item, StreamedStr):
        for chunk in item:
            # print the chunks as they are received
            print(chunk, sep="", end="")
        print()
    if isinstance(item, FunctionCall):
        # print the function call, then call it and print the result
        print(item)
        print(item())

# Hello! I'll get the weather for Cape Town and San Francisco for you.
# FunctionCall(<function get_weather at 0x1109825c0>, 'Cape Town')
# The weather in Cape Town is 20°C.
# FunctionCall(<function get_weather at 0x1109825c0>, 'San Francisco')
# The weather in San Francisco is 20°C.

jackmpcollins avatar Nov 30 '24 09:11 jackmpcollins