langchain
langchain copied to clipboard
Published
20 hours ago •
langchain-ai
langchain-ai
Agents can't parse OpenAIChat output
I'm on v0.0.98 and python3.10, and I'm getting this error when trying to use a zero-shot-react-description agent with OpenAIChat:
llm = OpenAIChat(temperature=0)
search = GoogleSearchAPIWrapper()
llm_math_chain = LLMMathChain(llm=llm, verbose=True)
tools = [
Tool(
name = "Search",
func=search.run,
description="useful for when you need to answer questions about current events"
),
Tool(
name="Calculator",
func=llm_math_chain.run,
description="useful for when you need to answer questions about math"
)
]
agent = initialize_agent(tools, llm, agent="zero-shot-react-description", verbose=True)
result = agent.run("What is OpenAI Foundry?")
> Entering new AgentExecutor chain...
I'm not sure what OpenAI Foundry is, so I should search for it.
Action: Search
Action Input: "OpenAI Foundry"
Observation: Feb 21, 2023 ... OpenAI has privately announced a new developer product called Foundry, which enables customers to run OpenAI model inference at scale w/ ... 2 days ago ... Conceptually, an industrial foundry is where businesses make the tools that make their physical products. OpenAI's Foundry will be where ... 1 day ago ... LLMs will soon replace lawyers and law enforcement. OpenAI DV could debate a supreme court level lawyer. reply ... Feb 21, 2023 ... OpenAI is reportedly launching a new developer platform, Foundry, that'll let customers run its models on dedicated compute infrastructure. 8 days ago ... Hey everyone, I've heard about this OpenAI's Foundry will let customers buy dedicated compute to run its AI models text-davinci-003 said I ... 2 days ago ... Conceptually, an industrial foundry is where businesses make the tools that make their physical products. OpenAI's Foundry will be where ... 8 days ago ... What Are OpenAI-Foundry Capabilities? · A static allocation of capacity dedicated to the user and providing a predictable environment that can be ... 6 days ago ... Well, well, well, what do we have here? OpenAI has gone and let slip some exciting details about their upcoming service Foundry, ... 6 days ago ... OpenAI is launching a developer platform known as Foundry to allow customers to run large artificial intelligence workloads based on its ... 8 days ago ... If the news posted on Twitter is trustworthy, then OpenAI's Foundry is made for “cutting-edge” customers running larger workloads. Users will be ...
Thought:
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
/var/folders/hz/pv3vmlvd52769n7ks7h17j1m0000gq/T/ipykernel_36393/1735989655.py in <cell line: 4>()
2 chat_start=time.time()
3 # with get_openai_callback() as cb:
----> 4 result = agent.run("What is OpenAI Foundry?")
5 # tokens = cb.total_tokens
6 end_time=time.time()
/opt/homebrew/lib/python3.10/site-packages/langchain/chains/base.py in run(self, *args, **kwargs)
237 if len(args) != 1:
238 raise ValueError("`run` supports only one positional argument.")
--> 239 return self(args[0])[self.output_keys[0]]
240
241 if kwargs and not args:
/opt/homebrew/lib/python3.10/site-packages/langchain/chains/base.py in __call__(self, inputs, return_only_outputs)
140 except (KeyboardInterrupt, Exception) as e:
141 self.callback_manager.on_chain_error(e, verbose=self.verbose)
--> 142 raise e
143 self.callback_manager.on_chain_end(outputs, verbose=self.verbose)
144 return self.prep_outputs(inputs, outputs, return_only_outputs)
/opt/homebrew/lib/python3.10/site-packages/langchain/chains/base.py in __call__(self, inputs, return_only_outputs)
137 )
138 try:
--> 139 outputs = self._call(inputs)
140 except (KeyboardInterrupt, Exception) as e:
141 self.callback_manager.on_chain_error(e, verbose=self.verbose)
/opt/homebrew/lib/python3.10/site-packages/langchain/agents/agent.py in _call(self, inputs)
501 # We now enter the agent loop (until it returns something).
502 while self._should_continue(iterations):
--> 503 next_step_output = self._take_next_step(
504 name_to_tool_map, color_mapping, inputs, intermediate_steps
505 )
/opt/homebrew/lib/python3.10/site-packages/langchain/agents/agent.py in _take_next_step(self, name_to_tool_map, color_mapping, inputs, intermediate_steps)
404 """
405 # Call the LLM to see what to do.
--> 406 output = self.agent.plan(intermediate_steps, **inputs)
407 # If the tool chosen is the finishing tool, then we end and return.
408 if isinstance(output, AgentFinish):
/opt/homebrew/lib/python3.10/site-packages/langchain/agents/agent.py in plan(self, intermediate_steps, **kwargs)
100 """
101 full_inputs = self.get_full_inputs(intermediate_steps, **kwargs)
--> 102 action = self._get_next_action(full_inputs)
103 if action.tool == self.finish_tool_name:
104 return AgentFinish({"output": action.tool_input}, action.log)
/opt/homebrew/lib/python3.10/site-packages/langchain/agents/agent.py in _get_next_action(self, full_inputs)
62 def _get_next_action(self, full_inputs: Dict[str, str]) -> AgentAction:
63 full_output = self.llm_chain.predict(**full_inputs)
---> 64 parsed_output = self._extract_tool_and_input(full_output)
65 while parsed_output is None:
66 full_output = self._fix_text(full_output)
/opt/homebrew/lib/python3.10/site-packages/langchain/agents/mrkl/base.py in _extract_tool_and_input(self, text)
136
137 def _extract_tool_and_input(self, text: str) -> Optional[Tuple[str, str]]:
--> 138 return get_action_and_input(text)
139
140
/opt/homebrew/lib/python3.10/site-packages/langchain/agents/mrkl/base.py in get_action_and_input(llm_output)
44 match = re.search(regex, llm_output, re.DOTALL)
45 if not match:
---> 46 raise ValueError(f"Could not parse LLM output: `{llm_output}`")
47 action = match.group(1).strip()
48 action_input = match.group(2)
ValueError: Could not parse LLM output: `Based on the search results, OpenAI Foundry is a new developer platform that allows customers to run OpenAI model inference at scale on dedicated compute infrastructure.
Action: None`
Getting the same error with Python 3.9, using OpenAIChat, and zero-shot-react-description on 0.0.98
OpenAIChat may not play nicely with current agent prompts
will experiment tmrw
I also encountered the same error. The version of LangCain is "0.0.101".
$ python custom_agent.py
You: Is there a tower taller than Tokyo Tower in Hokkaido?
> Entering new AgentExecutor chain...
Thought: Okay, let's think step by step. First, find out how tall the Tokyo Tower is. Then find out the tallest tower in Hokkaido. Then we can compare them and come up with an answer.
Thought: Do I need to use a tool? Yes
Action: Search
Action Input: Height of Tokyo Tower
Observation: 1,092′
Thought:Traceback (most recent call last):
File "/Users/fujiko/project/nakamurau1/llm_sandbox/custom_agent.py", line 83, in <module>
response = respond(user_input)
File "/Users/fujiko/project/nakamurau1/llm_sandbox/custom_agent.py", line 72, in respond
return agent_executor.run(input=input, conversation_history="")
File "/Users/fujiko/opt/anaconda3/lib/python3.9/site-packages/langchain/chains/base.py", line 268, in run
return self(kwargs)[self.output_keys[0]]
File "/Users/fujiko/opt/anaconda3/lib/python3.9/site-packages/langchain/chains/base.py", line 168, in __call__
raise e
File "/Users/fujiko/opt/anaconda3/lib/python3.9/site-packages/langchain/chains/base.py", line 165, in __call__
outputs = self._call(inputs)
File "/Users/fujiko/opt/anaconda3/lib/python3.9/site-packages/langchain/agents/agent.py", line 503, in _call
next_step_output = self._take_next_step(
File "/Users/fujiko/opt/anaconda3/lib/python3.9/site-packages/langchain/agents/agent.py", line 406, in _take_next_step
output = self.agent.plan(intermediate_steps, **inputs)
File "/Users/fujiko/opt/anaconda3/lib/python3.9/site-packages/langchain/agents/agent.py", line 102, in plan
action = self._get_next_action(full_inputs)
File "/Users/fujiko/opt/anaconda3/lib/python3.9/site-packages/langchain/agents/agent.py", line 64, in _get_next_action
parsed_output = self._extract_tool_and_input(full_output)
File "/Users/fujiko/opt/anaconda3/lib/python3.9/site-packages/langchain/agents/conversational/base.py", line 84, in _extract_tool_and_input
raise ValueError(f"Could not parse LLM output: `{llm_output}`")
ValueError: Could not parse LLM output: `Do I need to use a tool? Yes`
import os
os.environ["LANGCHAIN_HANDLER"] = "langchain"
from langchain.agents import ConversationalAgent, AgentExecutor
from langchain import LLMChain
from langchain.llms import OpenAIChat
from langchain.agents import load_tools
llm=OpenAIChat(temperature=0)
tools = load_tools(["serpapi", "llm-math"], llm=llm)
prefix = """Assistant is a large language model trained by OpenAI.
Assistant is designed to support a wide range of tasks, from answering simple questions to providing detailed explanations and discussions on a wide range of topics. As a language model, Assistant can generate human-like text based on input received, and can provide natural-sounding conversation or consistent, on-topic responses.
Assistant is constantly learning and improving, and its capabilities are always evolving. It can process vast amounts of text to understand and provide accurate and helpful answers to a variety of questions. Additionally, Assistant can generate its own text based on received input, allowing it to participate in discussions on a variety of topics, or provide explanations and commentary.
Overall, Assistant is a powerful tool that can support a variety of tasks and provide valuable insights and information on a wide range of topics. Whether you need help with a specific question or want to have a conversation about a specific topic, Assistant is here to help.
TOOLS:
------
Assistant has access to the following tools."""
suffix = """Answer the questions you know to the best of your knowledge. However, for complex questions outside of your knowledge, please think step by step and answer as follows.
====
User Input: Is there a tower taller than Tokyo Tower in Hokkaido?
Thought: Okay, let's think step by step. First, find out how tall the Tokyo Tower is. Then find out the tallest tower in Hokkaido. Then we can compare them and come up with an answer.
Thought: Do I need to use a tool? Yes
Action: Search
Action Input: Height of Tokyo Tower
Observation: [Result of searching the height of Tokyo Tower].
Thought: Do I need to use a tool? Yes
Action: Search
Action Input: Height of the tallest tower in Hokkaido
Observation: [Search result for the height of the tallest tower in Hokkaido].
Thought: Tokyo Tower is ****m high, and the tallest tower in Hokkaido is ****m high. Therefore, there is no tower taller than Tokyo Tower in Hokkaido.
AI: No, there is no tower taller than Tokyo Tower in Hokkaido.
====
Begin!
Previous conversation history:
{conversation_history}
User Input: {input}
{agent_scratchpad}"""
prompt = ConversationalAgent.create_prompt(
tools,
prefix=prefix,
suffix=suffix,
input_variables=["input", "conversation_history" ,"agent_scratchpad"]
)
llm_chain = LLMChain(llm=llm, prompt=prompt)
# tool_names = [tool.name for tool in tools]
agent = ConversationalAgent(llm_chain=llm_chain) #, allowed_tools=tool_names)
agent_executor = AgentExecutor.from_agent_and_tools(agent=agent, tools=tools, verbose=True)
def respond(input):
return agent_executor.run(input=input, conversation_history="")
while True:
user_input = input("You: ")
if user_input == "quit":
break
response = respond(user_input)
print("Chatbot:", response)
I have received a similar error with v0.0.103
Here is the code:
search = SerpAPIWrapper()
tools = [
Tool(
name = "Search",
func=search.run,
description="useful for when you need to answer questions about current events"
)
]
prefix = """You are a chatbot. Your directive is to help provide useful information to enable research and education. \
You have access to the following tools:"""
suffix = """Begin!"""
prompt = ZeroShotAgent.create_prompt(
tools,
prefix=prefix,
suffix=suffix,
input_variables=[]
)
messages = [
SystemMessagePromptTemplate(prompt=prompt),
HumanMessagePromptTemplate.from_template("{input}\n\nThis was your previous work "
f"(but I haven't seen any of it! I only see what "
"you return as final answer):\n{agent_scratchpad}")
]
prompt = ChatPromptTemplate.from_messages(messages)
llm_chain = LLMChain(llm=ChatOpenAI(temperature=0), prompt=prompt)
tool_names = [tool.name for tool in tools]
agent = ZeroShotAgent(llm_chain=llm_chain, allowed_tools=tool_names)
agent_executor = AgentExecutor.from_agent_and_tools(agent=agent, tools=tools, verbose=True)
response = agent_executor.run(message)
And here is the resulting error:
127.0.0.1 - - [07/Mar/2023 21:54:37] "OPTIONS /chat HTTP/1.1" 200 -
Message received: Please start a conversation with me!
> Entering new AgentExecutor chain...
127.0.0.1 - - [07/Mar/2023 21:54:39] "POST /chat HTTP/1.1" 500 -
Traceback (most recent call last):
File "/Users/reedbender/.pyenv/versions/3.9.15/lib/python3.9/site-packages/flask/app.py", line 2548, in __call__
return self.wsgi_app(environ, start_response)
File "/Users/reedbender/.pyenv/versions/3.9.15/lib/python3.9/site-packages/flask/app.py", line 2528, in wsgi_app
response = self.handle_exception(e)
File "/Users/reedbender/.pyenv/versions/3.9.15/lib/python3.9/site-packages/flask_cors/extension.py", line 165, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
File "/Users/reedbender/.pyenv/versions/3.9.15/lib/python3.9/site-packages/flask/app.py", line 2525, in wsgi_app
response = self.full_dispatch_request()
File "/Users/reedbender/.pyenv/versions/3.9.15/lib/python3.9/site-packages/flask/app.py", line 1822, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/Users/reedbender/.pyenv/versions/3.9.15/lib/python3.9/site-packages/flask_cors/extension.py", line 165, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
File "/Users/reedbender/.pyenv/versions/3.9.15/lib/python3.9/site-packages/flask/app.py", line 1820, in full_dispatch_request
rv = self.dispatch_request()
File "/Users/reedbender/.pyenv/versions/3.9.15/lib/python3.9/site-packages/flask/app.py", line 1796, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
File "/Users/reedbender/Repositories/Praxis/pria/app.py", line 78, in chat
response = agent_executor.run(message)
File "/Users/reedbender/.pyenv/versions/3.9.15/lib/python3.9/site-packages/langchain/chains/base.py", line 213, in run
return self(args[0])[self.output_keys[0]]
File "/Users/reedbender/.pyenv/versions/3.9.15/lib/python3.9/site-packages/langchain/chains/base.py", line 116, in __call__
raise e
File "/Users/reedbender/.pyenv/versions/3.9.15/lib/python3.9/site-packages/langchain/chains/base.py", line 113, in __call__
outputs = self._call(inputs)
File "/Users/reedbender/.pyenv/versions/3.9.15/lib/python3.9/site-packages/langchain/agents/agent.py", line 503, in _call
next_step_output = self._take_next_step(
File "/Users/reedbender/.pyenv/versions/3.9.15/lib/python3.9/site-packages/langchain/agents/agent.py", line 406, in _take_next_step
output = self.agent.plan(intermediate_steps, **inputs)
File "/Users/reedbender/.pyenv/versions/3.9.15/lib/python3.9/site-packages/langchain/agents/agent.py", line 102, in plan
action = self._get_next_action(full_inputs)
File "/Users/reedbender/.pyenv/versions/3.9.15/lib/python3.9/site-packages/langchain/agents/agent.py", line 64, in _get_next_action
parsed_output = self._extract_tool_and_input(full_output)
File "/Users/reedbender/.pyenv/versions/3.9.15/lib/python3.9/site-packages/langchain/agents/mrkl/base.py", line 138, in _extract_tool_and_input
return get_action_and_input(text)
File "/Users/reedbender/.pyenv/versions/3.9.15/lib/python3.9/site-packages/langchain/agents/mrkl/base.py", line 46, in get_action_and_input
raise ValueError(f"Could not parse LLM output: `{llm_output}`")
ValueError: Could not parse LLM output: `Sure, I'd be happy to chat with you! Is there anything specific you'd like to talk about or any questions you have?`
Thanks for the development y'all are doing!
