add_function_to_prompt bug fix

[phact]

This blows up when there's no "functions" in the dictionary even when tools is present because the inner function executes regardless (does not short circuit).

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
litellm	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 7, 2024 6:59pm

[phact]

Added unit test [and uncovered another bug]

Thanks!

[phact]

I'm hitting RuntimeError('dictionary changed size during iteration') in some cases, is wrapping

     for k in passed_params.keys():

is list() the right fix?

It gets me past the error but eventually I hit ValueError('Circular reference detected') on json.dumps on the actual completion call.

[phact]

Hmm this situation reproduces on main with the existing cohere unit test -- test_cohere_completion:

test_cohere_completion.py:20 (test_chat_completion_cohere)
def test_chat_completion_cohere():
        try:
            litellm.set_verbose = True
            messages = [
                {
                    "role": "user",
                    "content": "Hey",
                },
            ]
>           response = completion(
                model="cohere_chat/command-r",
                messages=messages,
                max_tokens=10,
            )

test_cohere_completion.py:30: 

                    extra_body[k] = passed_params[k]
            optional_params["extra_body"] = extra_body
        else:
            # if user passed in non-default kwargs for specific providers/models, pass them along
>           for k in passed_params.keys():
E           RuntimeError: dictionary changed size during iteration

../utils.py:5772: RuntimeError

During handling of the above exception, another exception occurred:

    def test_chat_completion_cohere_tool_calling():
        try:
            litellm.set_verbose = True
            messages = [
                {
                    "role": "user",
                    "content": "What is the weather like in Boston?",
                },
            ]
            response = completion(
                model="cohere_chat/command-r",
                messages=messages,
                tools=[
                    {
                        "type": "function",
                        "function": {
                            "name": "get_current_weather",
                            "description": "Get the current weather in a given location",
                            "parameters": {
                                "type": "object",
                                "properties": {
                                    "location": {
                                        "type": "string",
                                        "description": "The city and state, e.g. San Francisco, CA",
                                    },
                                    "unit": {
                                        "type": "string",
                                        "enum": ["celsius", "fahrenheit"],
                                    },
                                },
                                "required": ["location"],
                            },
                        },
                    }
                ],
            )
            print(response)
        except Exception as e:
>           pytest.fail(f"Error occurred: {e}")
E           Failed: Error occurred: dictionary changed size during iteration

test_cohere_completion.py:100: Failed

Do these get run in CI anywhere?

[phact]

passed_params = locals().copy()

seems to fix it but locals() is used all over the code base, wondering if this is going to lead to similar issues elsewhere.

[phact]

Tested this locally and it looks good.

@ishaan-jaff from what I can tell this is affecting completions that use tools for every model including openai even without litellm.add_function_to_prompt=True

Can we cut a release?

[ishaan-jaff]

from what I can tell this is affecting completions that use tools for every model including openai

what exactly is the issue you would see on OpenAI + tool usage ?

[phact]

E Failed: Error occurred: dictionary changed size during iteration

[phact]

I'll try to repro in a unit test

[ishaan-jaff]

'll try to repro in a unit test

would be great if you could add this test in this PR too thanks @phact

[phact]

Ran out of time yesterday, I'll look into adding another test in a new PR when I get a chance.