Fix Anthropic Messages Prompt Template function to add a third condition: list of text-content dictionaries

[eercanayar]

anthropic_messages_pt() function assumes that the assistant message's content is either string or none.

https://github.com/BerriAI/litellm/blob/006ea0abe030df162d6d0ad7ea0f86a89df8c970/litellm/llms/prompt_templates/factory.py#L686-L689

However there is a third condition possible, which is list of text-content dictionaries. This PR fixes this.

What's changed

Current implementation leads conversion in the anthropic_messages_pt() function from this:

[{'role': 'user', 'content': [{'type': 'text', 'text': 'hello'}]}, {'role': 'assistant', 'content': [{'type': 'text', 'text': 'Hello! How can I assist you today?'}]}, {'role': 'user', 'content': [{'type': 'text', 'text': 'hello again!'}]}]

to this:

[{'role': 'user', 'content': [{'type': 'text', 'text': 'hello'}]}, {'role': 'assistant', 'content': [{'type': 'text', 'text': [{'type': 'text', 'text': 'Hello! How can I assist you today?'}]}]}, {'role': 'user', 'content': [{'type': 'text', 'text': 'hello again!'}]}]

This message schema is clearly problematic and causes Anthropic API calls to fail as follows:

Request Sent from LiteLLM:

                response = client.invoke_model_with_response_stream(
                    body={"messages": [{"role": "user", "content": [{"type": "text", "text": "hello"}]}, {"role": "assistant", "content": [{"type": "text", "text": [{"type": "text", "text": "Hello! How can I assist you today?"}]}]}, {"role": "user", "content": [{"type": "text", "text": "hello again!"}]}], "max_tokens": 1024, "anthropic_version": "bedrock-2023-05-31"},
                    modelId=anthropic.claude-3-sonnet-20240229-v1:0,
                    accept=accept,
                    contentType=contentType
                )
                

RAW RESPONSE:
<litellm.utils.CustomStreamWrapper object at 0x111c23d90>


INFO:     127.0.0.1:64416 - "POST /chat/completions HTTP/1.1" 200 OK
Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.11/site-packages/litellm/proxy/proxy_server.py", line 2717, in async_data_generator
    async for chunk in response:
  File "/opt/homebrew/lib/python3.11/site-packages/litellm/utils.py", line 9864, in __anext__
    raise e
  File "/opt/homebrew/lib/python3.11/site-packages/litellm/utils.py", line 9796, in __anext__
    chunk = next(self.completion_stream)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/botocore/eventstream.py", line 603, in __iter__
    parsed_event = self._parse_event(event)
                   ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/botocore/eventstream.py", line 619, in _parse_event
    raise EventStreamError(parsed_response, self._operation_name)
botocore.exceptions.EventStreamError: An error occurred (validationException) when calling the InvokeModelWithResponseStream operation: messages.1.content.0.text.text: Input should be a valid string

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
litellm	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Apr 27, 2024 10:28am

[krrishdholakia]

Hey @eercanayar can you share a test input to litellm to repro this issue? Will add it to our ci/cd - https://github.com/BerriAI/litellm/blob/cdae08f3c30aeb83bff76bbe80c4584a325dbef5/litellm/tests/test_completion.py#L105

[krrishdholakia]

bump on this? @eercanayar

[eercanayar]

@krrishdholakia To ensure I understand correctly: I will create a new function (something like test_completion_with_text_content_dictionaries_claude_3() ) in the litellm/litellm/tests/test_completion.py file. Is that all I need to do? Or do I also need to reference that newly created function somewhere to enable it to run during the tests?

[krrishdholakia]

Yep - @eercanayar just add a test in there. As long as it starts with test_, it should get picked up by pytest

[eercanayar]

Thanks for the clarification. I will have time to finish this up by the end of the day.

[krrishdholakia]

Thanks @eercanayar excited to merge this

[eercanayar]

Implemented the test, fyi @krrishdholakia

[krrishdholakia]

Hey @eercanayar reviewed the openai spec - assistant_content.append({"type": "text", "text": assistant_text})

Assistant message content is not a list. Just a string.

[krrishdholakia]

Reverting this PR for now @eercanayar, If there is documentation / examples you can share of calling openai with a list of assistant messages - it would help.

[eercanayar]

Yes, it's also possible have it as a list. Some clients make calls in this format. (I'm not exactly sure why.)

Check out this example:

https://github.com/openai/openai-openapi/blob/a6aaddae428723201f8e295443332deff2a5c3ce/openapi.yaml#L145-L152

[krrishdholakia]

Hey @eercanayar reading the example,

the message is a user message, not an assistant message

https://github.com/openai/openai-openapi/blob/a6aaddae428723201f8e295443332deff2a5c3ce/openapi.yaml#L147

The PR you made + test included, seems to deal with the assistant message being a list.

[eercanayar]

Yes, that's correct. There is no example provided with a list of assistant messages in text-content dictionaries. Again, some clients implement calls in this way, and OpenAI API responds to them, and I don't know the exact reason. More openly, if the litellm package claims to function as an OpenAI proxy server, then it should support this request format. Because some very popular clients implement it this way (I can't criticize), and OpenAI servers respond to it.

Please check the implementation from continuedev/continue (11K stars.)

https://github.com/continuedev/continue/blob/3e68fd6d5f2be46c47e70d09aa1f83efc05a82ec/core/llm/constructMessages.ts

    content = [...ctxItems, ...content];

    msgs.push({
      role: historyItem.message.role,
      content,
    });

This implementation wraps every message.content as a list, regardless of the role.

Please check the example with the native OpenAI Python package:

from openai import OpenAI

client = OpenAI(
    api_key=API_KEY,
)

client.chat.completions.create(
    messages=[
        {
            'role': 'user',
            'content': [
                {
                    'type': 'text',
                    'text': 'hello'
                }
            ]
        },
        {
            'role': 'assistant',
            'content': [
                {
                    'type': 'text',
                    'text': 'Hello! How can I assist you today?'
                }
            ]
        },
        {
            'role': 'user',
            'content': [
                {
                    'type': 'text',
                    'text': 'hello again!'
                }
            ]
        }
    ],
    model="gpt-3.5-turbo",
)

Response:

ChatCompletion(id='chatcmpl-9J2sCq869XGwlQ1ClABWMagihP7FV', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='Hello again! How can I help you today?', role='assistant', function_call=None, tool_calls=None))], created=1714326304, model='gpt-3.5-turbo-0125', object='chat.completion', system_fingerprint='fp_3b956da36b', usage=CompletionUsage(completion_tokens=10, prompt_tokens=28, total_tokens=38))

This is why merging this is important; it will allow continuedev/continue users to use litellm to proxy calls to Anthropic Claude.

[eercanayar]

@krrishdholakia any insights about this?