llama_index Trying to Create ReACT Agent using Mistral LLM using HuggingFace Inference API but getting error: NotImplementedError: Messages passed in must be of odd length.

Question Validation

[X] I have searched both the documentation and discord for an answer.

Question

Trying to Create ReACT Agent using Mistral LLM using HuggingFace Inference API but getting error

HuggingFaceAPI import os from getpass import getpass from huggingface_hub import login

HuggingFace API HF_TOKEN = getpass()

login(token=HF_TOKEN)

Loading LLM endpoint from llama_index.llms.huggingface import HuggingFaceInferenceAPI llm = HuggingFaceInferenceAPI(model_name="mistralai/Mistral-7B-Instruct-v0.2", token=HF_TOKEN, num_output=512, is_function_calling_model=True, is_chat_model=True)

Simple Calculator from llama_index.core.agent import ReActAgent from llama_index.core.llms import ChatMessage from llama_index.core.tools import BaseTool, FunctionTool

Define Tool def multiply(a: int, b: int) -> int: """Multiply two integers and returns the result integer""" return a * b

def add(a: int, b: int) -> int: """Add two integers and returns the result integer""" return a + b

def subtract(a: int, b: int) -> int: """Subtract two integers and returns the result integer""" return a - b

multiply_tool = FunctionTool.from_defaults(fn=multiply) add_tool = FunctionTool.from_defaults(fn=add) subtract_tool = FunctionTool.from_defaults(fn=subtract)

Mistral agent = ReActAgent.from_tools([multiply_tool, add_tool, subtract_tool], llm=llm, verbose=True) response = agent.chat("What is 20+(2*4)? Calculate step by step.")

Error - NotImplementedError: Messages passed in must be of odd length.

May 06 '24 12:05 AnandUgale

The error you're encountering is due to the sequence of messages passed into the LlamaIndex framework not having an odd length, as it expects a pattern starting with a user message, alternating between user and assistant messages, and ending with a user message. To fix this, ensure your sequence of messages begins with a user message and follows this alternating pattern, maintaining an odd number of messages overall.

For example, you can prepend an initial system message to your sequence to adjust the number of messages:

initial_message = "Starting calculation..."
response = agent.chat([initial_message, "What is 20+(2*4)? Calculate step by step."])

This adjustment ensures the sequence has an odd number of messages, aligning with the framework's expectations and resolving the NotImplementedError you're facing.

Details

Sources

About Dosu This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

May 06 '24 14:05 dosubot[bot]

Yea huggingface does not like system messages from some LLMs

You'll have to write your own messages_to_prompt() function hook to transform message objects into a prompt for your LLM

Here's a naive way for mistral

def messages_to_prompt(messages):
  message_str = "\n".join([str(x) for x in messages])
  return f"[INST] {message_str} [/INST] "
  
llm = HuggingFaceLLM(..., messages_to_prompt=messages_to_prompt)

May 06 '24 15:05 logan-markewich

The error you're encountering is due to the sequence of messages passed into the LlamaIndex framework not having an odd length, as it expects a pattern starting with a user message, alternating between user and assistant messages, and ending with a user message. To fix this, ensure your sequence of messages begins with a user message and follows this alternating pattern, maintaining an odd number of messages overall.

For example, you can prepend an initial system message to your sequence to adjust the number of messages:
initial_message = "Starting calculation..."
response = agent.chat([initial_message, "What is 20+(2*4)? Calculate step by step."])
This adjustment ensures the sequence has an odd number of messages, aligning with the framework's expectations and resolving the NotImplementedError you're facing.

Details

To continue the conversation, mention @dosu.

Tried your suggestion but encountering another error.

initial_message = "Starting calculation..." response = agent.chat([initial_message, "What is 20+(2*4)? Calculate step by step."]) print(str(response))

error - ValidationError: 1 validation error for Task input str type expected (type=type_error.str)

May 06 '24 17:05 AnandUgale

To resolve the ValidationError: str type expected error when using the agent.chat method, it's crucial to ensure that the input is formatted as a list of ChatMessage objects, each with a specified role and content. This is because the agent.chat method expects input in a specific structure that includes the role of each message (e.g., user, system) and the content as a string.

Here's how you can correctly format your input:

Prepare Chat Messages: Create ChatMessage objects for each message you want to send to the agent. Each message should have a role (e.g., MessageRole.USER for user messages) and a content (the message text).
Use ReActChatFormatter for Formatting: If you're using a custom formatting approach, ensure it aligns with the expected input structure of the agent.chat method. The ReActChatFormatter class can help format the chat history into the required format.
Example Code:

from llama_index.legacy.core.llms.types import ChatMessage, MessageRole

# Example user query
user_query = "What is 20+(2*4)? Calculate step by step."

# Create a ChatMessage object for the user query
chat_history = [ChatMessage(role=MessageRole.USER, content=user_query)]

# Assuming `agent` is your ReActAgent instance
response = agent.chat(chat_history)
print(str(response))

This approach ensures that each message passed to the agent.chat method is correctly typed and structured, thus avoiding the ValidationError. Remember, the key is to use ChatMessage objects with the appropriate role and content for each message in your chat history.

Details

Sources

About Dosu This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

May 06 '24 19:05 dosubot[bot]

Yea huggingface does not like system messages from some LLMs

You'll have to write your own messages_to_prompt() function hook to transform message objects into a prompt for your LLM

Here's a naive way for mistral
def messages_to_prompt(messages):
  message_str = "\n".join([str(x) for x in messages])
  return f"[INST] {message_str} [/INST] "
  
llm = HuggingFaceLLM(..., messages_to_prompt=messages_to_prompt)

I am trying to access it using the HuggingFace Inference API, not with HuggingFace LLM. Can you tell me how it could be possible using the HuggingFace Inference API?

May 08 '24 16:05 AnandUgale

@AnandUgale agent.chat() takes a single string, not a list. Dosu be crazy.

The suggestion I gave above should work

May 09 '24 01:05 logan-markewich

This does not solve the issue for me. I tried using the callback with HuggingFaceInferenceAPI, however I am still getting the same error: Messages passed in must be of odd length. I noticed that @logan-markewich proposed a solution using HuggingFaceLLM, however OP mentioned HuggingFaceInferenceAPI.

I noticed that it is possible to avoid the error and query for text_generation by setting the task field to anything besides conversational:

self.llm_client = HuggingFaceInferenceAPI(
    model_name=model_id,
    timeout=120,
    messages_to_prompt=messages_to_prompt,
    task='complete'
)

Is there any way to get conversational style chat working?

Jun 16 '24 17:06 atomwalk12

llama_index llama_index copied to clipboard

Trying to Create ReACT Agent using Mistral LLM using HuggingFace Inference API but getting error: NotImplementedError: Messages passed in must be of odd length.

Question Validation

Question

Details

Details

Details

llama_index
llama_index copied to clipboard