(feat) allow users to opt out of message merge - Anthropic

[ishaan-jaff]

Addressing https://github.com/BerriAI/litellm/discussions/2664

Problem - LiteLLM will format messages to Anthropic to ensure the call does not drop. We (LiteLLM) believe default behavior should be the call should not fail/get rejected BUT a user should be able to opt out of LiteLLM applying any type of prompt formatting

Usage

LiteLLM will not edit / re-order the message to anthropic

litellm.disable_message_merge == True

    """
    format messages for anthropic
    1. Anthropic supports roles like "user" and "assistant", (here litellm translates system-> assistant)
    2. The first message always needs to be of role "user"
    3. Each message must alternate between "user" and "assistant" (this is not addressed as now by litellm)
    4. final assistant content cannot end with trailing whitespace (anthropic raises an error otherwise)
    5. System messages are a separate param to the Messages API (used for tool calling)
    6. Ensure we only accept role, content. (message.name is not supported)
    """

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
litellm	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Mar 24, 2024 3:01am

[ishaan-jaff]

cc @morgante

[morgante]

Thanks for working on this.

Why not use the existing modify_params setting though? In general I suggest trying to avoid settings bloat.

LiteLLM will format messages to Anthropic to ensure the call does not drop. We (LiteLLM) believe default behavior should be the call should not fail/get rejected

I think there's a misunderstanding here. I don't really care what you do with poorly formatted cases - my point is that you shouldn't have a complex loop where you rewrite messages messages that are already formatted just fine.