langchain
langchain copied to clipboard
langchain-ai
ValueError: Could not parse LLM output:
`agent_chain = initialize_agent( tools=tools, llm= HuggingFaceHub(repo_id="google/flan-t5-xl"), agent="conversational-react-description", memory=memory, verbose=False)
agent_chain.run("Hi")`
throws error. This happens with Bloom as well. Agent only with OpenAI is only working well.
`_(self, inputs, return_only_outputs) 140 except (KeyboardInterrupt, Exception) as e: 141 self.callback_manager.on_chain_error(e, verbose=self.verbose) --> 142 raise e 143 self.callback_manager.on_chain_end(outputs, verbose=self.verbose) ... ---> 83 raise ValueError(f"Could not parse LLM output: "{llm_output}") 84 action = match.group(1) 85 action_input = match.group(2)
ValueError: Could not parse LLM output: Assistant, how can I help you today?`
In your case google/flan-t5-xl does not follow the conversational-react-description template.
The LLM should output:
Thought: Do I need to use a tool? Yes
Action: the action to take, should be one of [{tool_names}]
Action Input: the input to the action
Observation: the result of the action
Thought: Do I need to use a tool? No
{ai_prefix}: [your response here]
For your example agent_chain.run("Hi") I suppose the agent should not use any tool. So conversational-react-description would look for the word {ai_prefix}: in the response, but when parsing the response it can not find it (and also there is no "Action").
I think this happens in these models because they are not trained to follow instructions, they are LLMs used for language modeling, but in the case of OpenAI GPT-3.5, it is specifically trained to follow user instructions (like asking it to output the format that I mentioned before, Thought, Action, Action Input, Observation or Thought, {ai_prefix})
I tested it, in my case, I got ValueError: Could not parse LLM output: 'Assistant, how can I help you today?'. So in here we were looking for {ai_prefix}:. Ideally the model should output Thought: Do I need to use a tool? No \nAI: how can I help you today? ({ai_prefix} in my example was "AI").
I hope this is clear!
Am having the same issue
My code is here
@st.cache_resource
def create_tool(_index, chosen_pdf):
tools = [
Tool(
name=f"{chosen_pdf} index",
func=lambda q: str(_index.query(q)),
description="Useful to answering questions about the given file",
return_direct=True,
),
]
return tools
@st.cache_resource
def create_agent(chosen_class, chosen_pdf):
memory = ConversationBufferMemory(memory_key="chat_history")
llm = OpenAI(temperature=0, model_name="gpt-3.5-turbo")
index = get_index(chosen_class, chosen_pdf)
tools = create_tool(index, chosen_pdf)
agent = initialize_agent(
tools, llm, agent="conversational-react-description", memory=memory
)
return agent
def query_gpt_memory(chosen_class, chosen_pdf, query):
agent = create_agent(chosen_class, chosen_pdf)
res = agent.run(input=query)
st.session_state.memory = agent.memory.buffer
return res
error output
File "/Users/benedictneo/fun/ClassGPT/app/utils.py", line 158, in query_gpt_memory
res = agent.run(input=query)
File "/Users/benedictneo/miniforge3/lib/python3.9/site-packages/langchain/chains/base.py", line 268, in run
return self(kwargs)[self.output_keys[0]]
File "/Users/benedictneo/miniforge3/lib/python3.9/site-packages/langchain/chains/base.py", line 168, in __call__
raise e
File "/Users/benedictneo/miniforge3/lib/python3.9/site-packages/langchain/chains/base.py", line 165, in __call__
outputs = self._call(inputs)
File "/Users/benedictneo/miniforge3/lib/python3.9/site-packages/langchain/agents/agent.py", line 503, in _call
next_step_output = self._take_next_step(
File "/Users/benedictneo/miniforge3/lib/python3.9/site-packages/langchain/agents/agent.py", line 406, in _take_next_step
output = self.agent.plan(intermediate_steps, **inputs)
File "/Users/benedictneo/miniforge3/lib/python3.9/site-packages/langchain/agents/agent.py", line 102, in plan
action = self._get_next_action(full_inputs)
File "/Users/benedictneo/miniforge3/lib/python3.9/site-packages/langchain/agents/agent.py", line 64, in _get_next_action
parsed_output = self._extract_tool_and_input(full_output)
File "/Users/benedictneo/miniforge3/lib/python3.9/site-packages/langchain/agents/conversational/base.py", line 84, in _extract_tool_and_input
raise ValueError(f"Could not parse LLM output: `{llm_output}`")
ValueError: Could not parse LLM output: `Thought: Do I need to use a tool? No
However, I still get a response even with this value error
Just to understand clearly, is there a correct way of going about this? I am also getting the same error even if i use the chat-* type agents.
Or should we wait for an update?
just boosting this thread -- this is a common issue in my builds with langchain, but I also utilize promptlayer, so I can see that the outputs are indeed parsable to the API that routes my convo logs...clearly something is wrong and I have yet to find a full-proof solution to avoiding this.
+1 This seems to a common issue with chat agents which are the need of the hour!
I was able to fix this locally by simply calling the LLM again when there is a Parse error. I am sure my code is not exactly in the spirit of langchain, but if anyone wants to take the time to review my branch https://github.com/tomsib2001/langchain/tree/fix_parse_LLL_error (it's a POC, not a finished MR) and tell me:
- either that there is a fundamental reason not to do things this way
- how to make my code more in line with the standards of langchain I would be grateful. I am happy to help as I'm getting a lot of value and fun from langchain, but I'm fairly new to contributing to large open source projects, so please bear with me.
I have a same problem. Don't know what can be the reason, when the output that cause the error contains well formulated answer.
This is super hacky, but while we don't have a solution for this issue, you can use this:
try:
response = agent_chain.run(input=query_str)
except ValueError as e:
response = str(e)
if not response.startswith("Could not parse LLM output: `"):
raise e
response = response.removeprefix("Could not parse LLM output: `").removesuffix("`")
Might be anecdotal, but I think GPT does better with JSON-type formatting, maybe that will help with the formating issues? I made a couple of changes on my local copy of langchain, seems to work much more reliably
@noobmaster19 can you share what changes you made to make things more reliable?
Are there any HuggingFace models that work as an agent, or are we forced to use OpenAI?
It is possible to pass output parser to the agent executor. Here is how I did:
class NewAgentOutputParser(BaseOutputParser):
def get_format_instructions(self) -> str:
return FORMAT_INSTRUCTIONS
def parse(self, text: str) -> Any:
print("-" * 20)
cleaned_output = text.strip()
# Regex patterns to match action and action_input
action_pattern = r'"action":\s*"([^"]*)"'
action_input_pattern = r'"action_input":\s*"([^"]*)"'
# Extracting first action and action_input values
action = re.search(action_pattern, cleaned_output)
action_input = re.search(action_input_pattern, cleaned_output)
if action:
action_value = action.group(1)
print(f"First Action: {action_value}")
else:
print("Action not found")
if action_input:
action_input_value = action_input.group(1)
print(f"First Action Input: {action_input_value}")
else:
print("Action Input not found")
print("-" * 20)
if action_value and action_input_value:
return {"action": action_value, "action_input": action_input_value}
# Problematic code left just in case
if "```json" in cleaned_output:
_, cleaned_output = cleaned_output.split("```json")
if "```" in cleaned_output:
cleaned_output, _ = cleaned_output.split("```")
if cleaned_output.startswith("```json"):
cleaned_output = cleaned_output[len("```json"):]
if cleaned_output.startswith("```"):
cleaned_output = cleaned_output[len("```"):]
if cleaned_output.endswith("```"):
cleaned_output = cleaned_output[: -len("```")]
cleaned_output = cleaned_output.strip()
response = json.loads(cleaned_output)
return {"action": response["action"], "action_input": response["action_input"]}
# end of problematic code
def make_chain():
memory = ConversationBufferMemory(
memory_key="chat_history", return_messages=True)
agent = ConversationalChatAgent.from_llm_and_tools(
llm=ChatOpenAI(), tools=[], system_message=SYSTEM_MESSAGE, memory=memory, verbose=True, output_parser=NewAgentOutputParser())
agent_chain = AgentExecutor.from_agent_and_tools(
agent=agent,
tools=tools,
memory=memory,
verbose=True,
)
return agent_chain
@Saik0s I am interested in this approach, can you explain what is the SYSTEM_MESSAGE, have you changed it? and can you show how the output looks? does it also work with zero-shot agent?
@gambastyle I did some modifications to system message, but that is not related to the problem. You can find default system message here. I just wanted to show my approach to fix problem by introducing upgraded output parser NewAgentOutputParser
Everyone seems to have missed klimentij's fix in this thread? It fixes it for me, even though the best solution would be to fix the agent to use a series of chains like he did privately.
Having a prompt doing more than one thing and parsing the output is a bad idea with LLMs, that is why chains exist and are not being properly used in this agent.
See code and comments here: https://github.com/hwchase17/langchain/pull/1707
@Saik0s did you check klimentij's parser changes and do you changes compare?
@tiagoefreitas I checked it, his approach is similar to what I did, and he did it directly in the library code. I would even say that his approach is better, because there is no regular expressions in it 😅
super duper hacky solution:
if __name__ == "__main__":
try:
response = agent.run(
"Could you fix this issue please?"
)
except Exception as e:
response = str(e)
if "Could not parse LLM output: `" not in response:
raise e
match = re.search(r"`(.*?)`", response)
if match:
last_output = match.group(1)
print("Last output:", last_output)
else:
print("No match found")
# Last output: hopefully it will be fixed soon!
try: response = agent_chain.run(input=query_str) except ValueError as e: response = str(e) if not response.startswith("Could not parse LLM output:
"): raise e response = response.removeprefix("Could not parse LLM output:").removesuffix("`")
Where do I paste this? To make the agent do this?
I am still getting the same error even after applying the proposed solution, can any one please help me?
I'm getting this whenever I try to add an ai_prefix to ConversationalAgent.create_prompt(). This one is really surprising to me. It's just changing out the string value of "AI"
def create_prompt(
cls,
tools: Sequence[BaseTool],
prefix: str = PREFIX,
suffix: str = SUFFIX,
format_instructions: str = FORMAT_INSTRUCTIONS,
ai_prefix: str = "AI",
human_prefix: str = "Human",
input_variables: Optional[List[str]] = None,
) -> PromptTemplate:
With the prefix:
prompt = ConversationalAgent.create_prompt(
tools,
prefix,
suffix,
ai_prefix="Dude",
input_variables=["input", "chat_history", "agent_scratchpad"],
)
Output
New Question: hello
Thought: Do I need to use a tool? No
Dude: Hello! How can I help you with your questions today?Traceback (most recent call last):
raise OutputParserException(f"Could not parse LLM output: `{text}`")
langchain.schema.OutputParserException: Could not parse LLM output: `Thought: Do I need to use a tool? No
Dude: Hello! How can I help you with your questions today?`
Without Prefix
New Question: hello
Thought: Do I need to use a tool? No
AI: Hello! How can I help you with your Medicare questions today?
in short there is not a single huggingface model, that is not throwing this error?
