AI Prefix and Human Prefix not correctly reflected in

[scoupleux]

 prompt = ChatPromptTemplate.from_messages([
        SystemMessagePromptTemplate.from_template(template=system_template),
        MessagesPlaceholder(variable_name="history"),
        HumanMessagePromptTemplate.from_template("{input}")
    ])

    llm = ChatOpenAI(temperature=0.9)
    memory = ConversationBufferMemory(return_messages=True, ai_prefix="SpongebobSquarePants", human_prefix="Bob")
    conversation = ConversationChain(memory = memory, prompt = prompt, llm = llm, verbose=True)

using ChatPromptTemplate.from_messages will later use the method get_buffer_string in the to_string() for the class ChatPromptValue in chat.py in Prompts. The format does not care of the new ai_prefix or human_prefix.

How can I change that ? Thanks

[Joshua-Shepherd]

They are hardcoded in memory/base.ts bufferMemory.ts and summary.ts modules and in the schema For some reason the prefix input variables aren't being passed along. You can either, Fix it and open a PR, or just manually edit them for your project.

I know you're using LC in Python but the same would apply

[Joshua-Shepherd]

Its supposed to be fixed.. But its not: https://python.langchain.com/en/latest/modules/memory/examples/conversational_customization.html#human-prefix

[dmyoung9]

I've found a partial solution for this here:

If you use ChatMessagePromptTemplate.from_template("{input}", role="Bob"), then the "current" messages will be prefixed properly. Unfortunately, this will also cause the history messages to say Bob: Bob: ... 🤬

My next attempt was setting human_prefix='', but that resulted in seeing : Bob: ..., so that's not quite it either 🤔

EDIT: For what it's worth, setting ai_prefix is working for me.

[dmyoung9]

@scoupleux It looks like the defaults actually lead to the history messages saying Human: Human: ... anyways... something feels off here, but I haven't gotten to the bottom of it yet.

[Joshua-Shepherd]

Removing the Role from the pushed string helped mitigate any issues with responses staying "In Character" and not being returned like "As and AI im blah blah blah"

[dmyoung9]

@Joshua-Shepherd Removing that (at least in the Python library, which this repository is for) causes there to be no role labels at all in the generated history messages.

The problem here seems to be that ChatPromptValue.to_string() prepends the role artificially, which probably shouldn't happen in this particular case.

[dmyoung9]

Also, I'm thinking https://github.com/hwchase17/langchain/blob/master/langchain/memory/chat_memory.py#L35 might have something to do with it as well... No matter the role you give your message, this will always create a HumanMessage.

[dmyoung9]

I've come up with a solution that works for arbitrary roles by using ChatMessage instead of HumanMessage:

from typing import Any

from langchain.memory import ConversationBufferMemory
from langchain.schema import ChatMessage

class RoleBasedConversationBufferMemory(ConversationBufferMemory):
    def save_context(self, inputs: dict[str, Any], outputs: dict[str, str]) -> None:
        """Save context from this conversation to buffer."""
        input_str, output_str = self._get_input_output(inputs, outputs)

        # take advantage of `get_buffer_string` prepending the role
        role, content = input_str.split(": ")[:2]
        self.chat_memory.add_message(ChatMessage(role=role, content=content))
        self.chat_memory.add_ai_message(output_str)

You should be able to use this like memory = RoleBasedConversationBufferMemory(ai_prefix="K.A.R.E.N."), and it should give consistent (and non-doubled) roles based on the messages in your template.

Take note, this does actually depend on you using ChatPromptValue.to_string() in the end before passing off to your chain, because it is taking advantage of the fact that get_buffer_string(...) prepends the role for you in order to split the role from the content.

I still think this is sort of a "bug"... it would probably be better to store and manipulate ChatMessages in memory as is, so that their roles can be more accurately determined after the fact. That would also allow for human_prefix and ai_prefix to be applied during the memory phase as well, which would likely solve this entirely.

[Avinash-Raj]

@dmyoung9 endup with the following exception..

openai.error.InvalidRequestError: 'Question' is not one of ['system', 'assistant', 'user', 'function'] - 'messages.3.role'

[dmyoung9]

@Avinash-Raj I've just updated to 0.0.208, and I'm not getting the exception.

I am using this as memory for a ChatOpenAI, and not with agents, if that makes a difference.

[bitnom]

https://github.com/hwchase17/langchain/issues/3234 is potentially related. This whole feature needs to be uniform across all these classes. It's no good this way.

[CarloNicolini]

I have found that to enable a good multi-turn conversation with Llama2-chat based models, one needs to modify the get_buffer_string function by overriding it with a custom one. Specifically the list of AIMessage, SystemMessage and HumanMessage messages must be converted not only by prefixing the role like Human: or AI: , but rather by wrapping it with specific tags.

In the Llama2-chat case the template looks like that (beware of all spaces and newlines between tags, they are important.

<s>[INST] <<SYS>>

You are are a helpful school teacher, able to explain complicated things to childrens.

<</SYS>>


Hi there! [/INST] Hello! How can I help you today? </s><s>[INST] Teach me quantum mechanics like I am 5. [/INST] Quantum mechanics is like a toy.... </s>

Here are some resources that confirm this necessity:

• Reddit-LocalLlama
• Another blog on Llama2-chat prompting
• oobabooga text-generation-webui template for Llama models

What are the best approaches to enable this wrapping of history messages based on specific model templates? One possibility, as far as I understand is to subclass the specific memory class like ConversationSummaryBufferMemory and override the save_context method with specific model based functions. Alternatively, a simpler option that can be extended to all subclasses of BaseChatMemory is to add as a base class member an optional callable get_buffer_string where the user transforms the list of messages in the context in the way he wants.

[ClementViricel]

Hello,

I have the same issue here :

memory = ConversationBufferMemory(
    memory_key="chat_history",
    input_key="question",
    output_key="output_text",
    return_messages=True,
    human_prefix="Humain",
    ai_prefix="Assistant",
)

Here the verbosity of the chain :

Human: Quels sont les traitements du cancer du sein ?
AI:  Les traitements du cancer du sein dépendent du type et du stade du cancer, ainsi que de l'âge et de l'état de santé général de la patiente. Les traitements peuvent inclure la chirurgie, la radiothérapie, la chimiothérapie, l'hormonothérapie et la thérapie ciblée.
La chirurgie est le traitement le plus courant du cancer du sein. Elle peut être utilisée pour enlever la tumeur, les ganglions lymphatiques voisins et/ou le sein entier.
La radiothérapie est utilisée pour tuer les cellules cancéreuses qui peuvent rester après la chirurgie. Elle peut être administrée par voie externe ou interne.
La chimiothérapie est utilisée pour tuer les cellules cancéreuses qui se sont propagées à d'autres parties du corps. Elle peut être administrée par voie orale ou intraveineuse.
L'hormonothérapie est utilisée pour traiter les cancers du sein qui sont sensibles aux hormones. Elle peut être administrée par voie orale ou par injection.
La thérapie ciblée est utilisée pour traiter les cancers du sein qui présentent des mutations génétiques spécifiques. Elle peut être administrée par voie orale ou intraveineuse.
Human:  Quels sont les symptômes du cancer du sein ?

You see that human and AI are hardcoded.

[amanmamgain9]

You can change like this for now general_summary_memory.ai_prefix = "Yumi" general_summary_memory.human_prefix = "Human"

[ChrisDeadman]

I second what @CarloNicolini said - it needs to be properly templated. Tried to use langchain today with a model that expects the ChatML format and found out that this is simply not possible since it's hardcoded in various places for Llama2 format 😩

[nickyeolk]

Fixing this wouldn't solve the issue for Llama 2 template or ChatML template. It requires more than a prefix, but the ability to add a suffix as well.

I believe a solution to the initial problem would involve adding human_prefix and ai_prefix as a class variable to HumanMessage and AIMessage, then passing the human_prefix and ai_prefix values to the messages when calling buffer_as_messages inside ConversationBufferMemory. It is already done for the buffer_as_str method.
That way your messages will end up looking like messages=[HumanMessage(content='human msg 1', human_prefix='new_human_prefix'), AIMessage(content=AI reply', ai_prefix='new_ai_prefix')]

You can then modify to_string to look for human_prefix and ai_prefix and modify arguments passed into get_buffer_string add **kwargs to to_string call in generate_prompt

[ChrisDeadman]

@nickyeolk But why use prefixes and suffixes if we have proper message templates already in langchain? I'd think that supplying the templates directly would be much more flexible and cleaner than defining templates internally that use those prefixes and suffixes or am I missing something here?

[nickyeolk]

My use case was that I was trying to use LLMChain with ChatPromptTemplate and ConversationBufferMemory like so:

prompt = ChatPromptTemplate.from_messages([
    SystemMessage(content="You are a helpful AI assistant that gives concise answers"),
    MessagesPlaceholder(variable_name="history_x"),
    ("human", "{human_input}")
])
memory = ConversationBufferMemory(memory_key='history_x', 
                                  human_prefix='human_special', 
                                  ai_prefix='ai_special', 
                                  return_messages=True)

You're probably right that a better way to do this would be to supply templates directly. I just thought using the ChatPromptTemplates and ConversationBufferMemory would save some time. But it seems that we are kind of 'forced' to use human message, ai message, and system messages as the default prefixes.

[dosubot[bot]]

Hi, @scoupleux,

I'm helping the LangChain team manage their backlog and am marking this issue as stale. From what I understand, you raised an issue regarding the incorrect reflection of AI and human prefixes in the conversation. The discussion in the comments includes various attempts and suggestions to address the issue, such as manually editing the prefixes, potential solutions involving role-based conversation memory, and the need for proper templating to handle different message formats. There are also mentions of related issues and potential solutions for specific model templates.

Could you please confirm if this issue is still relevant to the latest version of the LangChain repository? If it is, please let the LangChain team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or the issue will be automatically closed in 7 days.

Thank you for your understanding and cooperation.

[mplaul]

This issue still exists and is not stale. I am using a DynamoDB to persist the chat history. Using this memory history class in combination with LCEL runnables (RunnableWithMessageHistory) and finally invoking a chain on Bedrock/Claude 2.1 throws ALTERNATION_ERRORs as generating and returning the chat prompts is using a default value ai_prefix: str = "AI" while Claude expects the prefix to be "Assistant". The ai_prefix is not exposed to developers and the quickest fix i found so far is to modify the default value in the function itself: https://github.com/langchain-ai/langchain/blob/master/libs/core/langchain_core/messages/utils.py#L23