Issue: openai functions agent does not respect tools and arguments

[falmanna]

Issue you'd like to raise.

When mixing gpt-3.5-turbo-0613, openai-functions agent, and PythonAstREPLTool tool, GPT3.5 stops respecting the tool name and the arguments hack introduced in the OpenAIFunctionsAgent.

The error log is:

Could not parse tool input: {'name': 'python', 'arguments': "len(cases_df['case_id'].unique())"} because the `arguments` is not valid JSON.

Which means the model isn't respecting the specs accurately. In my case, the confusion was always that the name of the tool is python instead of python_repl_ast, and the arguments is the actual python code instead of the requested obj format with __arg1 attr.

Suggestion:

I temporarily fixed it by 1- extending the OpenAIFunctionsAgent and overriding the _parse_ai_message to handle arguments confusion. 2- extending the PythonAstREPLTool and altering its name and description a bit.

class CustomPythonAstREPLTool(PythonAstREPLTool):
    name = "python"
    description = (
        "A Python shell. Use this to execute python commands. "
        "The input must be an object as follows: "
        "{'__arg1': 'a valid python command.'} "
        "When using this tool, sometimes output is abbreviated - "
        "Make sure it does not look abbreviated before using it in your answer. "
        "Don't add comments to your python code."
    )

def _parse_ai_message(message: BaseMessage) -> Union[AgentAction, AgentFinish]:
    """Parse an AI message."""
    if not isinstance(message, AIMessage):
        raise TypeError(f"Expected an AI message got {type(message)}")

    function_call = message.additional_kwargs.get("function_call", {})

    if function_call:
        function_call = message.additional_kwargs["function_call"]
        function_name = function_call["name"]
        try:
            _tool_input = json.loads(function_call["arguments"])
        except JSONDecodeError:
            print(
                f"Could not parse tool input: {function_call} because "
                f"the `arguments` is not valid JSON."
            )
            _tool_input = function_call["arguments"]

        # HACK HACK HACK:
        # The code that encodes tool input into Open AI uses a special variable
        # name called `__arg1` to handle old style tools that do not expose a
        # schema and expect a single string argument as an input.
        # We unpack the argument here if it exists.
        # Open AI does not support passing in a JSON array as an argument.
        if "__arg1" in _tool_input:
            tool_input = _tool_input["__arg1"]
        else:
            tool_input = _tool_input

        content_msg = "responded: {content}\n" if message.content else "\n"

        return _FunctionsAgentAction(
            tool=function_name,
            tool_input=tool_input,
            log=f"\nInvoking: `{function_name}` with `{tool_input}`\n{content_msg}\n",
            message_log=[message],
        )

    return AgentFinish(return_values={"output": message.content}, log=message.content)

class CustomOpenAIFunctionsAgent(OpenAIFunctionsAgent):
    def plan(
        self,
        intermediate_steps: List[Tuple[AgentAction, str]],
        callbacks: Callbacks = None,
        **kwargs: Any,
    ) -> Union[AgentAction, AgentFinish]:
        """Given input, decided what to do.
        Args:
            intermediate_steps: Steps the LLM has taken to date, along with observations
            **kwargs: User inputs.
        Returns:
            Action specifying what tool to use.
        """
        user_input = kwargs["input"]
        agent_scratchpad = _format_intermediate_steps(intermediate_steps)
        prompt = self.prompt.format_prompt(
            input=user_input, agent_scratchpad=agent_scratchpad
        )
        messages = prompt.to_messages()
        predicted_message = self.llm.predict_messages(
            messages, functions=self.functions, callbacks=callbacks
        )
        agent_decision = _parse_ai_message(predicted_message)
        return agent_decision

Not sure if this will be improved on the API level, but it is worth looking at it. Improving the fake arguments' names and tools names might improve this as it seems related to the issue.

[francisjervis]

It's a general performance issue with the model - it hallucinates and ignores instructions like enums. Using natural language or sudocode prompts to return JSON gives consistently better results. The solution is not to use functions.

[falmanna]

It's a general performance issue with the model - it hallucinates and ignores instructions like enums. Using natural language or sudocode prompts to return JSON gives consistently better results. The solution is not to use functions.

This is really interesting because it is supposed to be the other way. This model was fine-tuned on returning the correct requested json, so its error rate should be lower than using just a prompt. I don't know what they are doing internally, but I think they are converting the functions list piece to a prompt in the same format they used to fine-tune the model, so it is really weird that we might get better results by crafting our own prompt instead of using functions.

This somehow makes sense as I think the model cares about the functions and the parameter names, hence the confusion it is making when calling back the python tool. I believe that improving this piece in langcahin might yield better results as I didn't have any issues since I implemented the changes above.

[li-xiaohui]

Hi @falmanna ,

Thanks a lot for sharing the issue and temp fix. I'm using create_pandas_dataframe_agent together with gpt-3.5-turbo, and keep getting this exact issue. I think this issue is still open as of v0.0.228. Would you care to share how I can apply your fix with create_pandas_dataframe_agent?

I call create_pandas_dataframe_agent with the following code:

agent = create_pandas_dataframe_agent(
    ChatOpenAI(temperature=0, model="gpt-3.5-turbo-0613"),
    df,
    verbose=True,
    agent_type=AgentType.OPENAI_FUNCTIONS,
)

Thank you very much in advance.

[falmanna]

hey @li-xiaohui

I had to initialize the agent manually, here is how.

dataset={"df": df}
tools = [CustomPythonAstREPLTool(locals=dataset)]
tool_names = [tool.name for tool in tools]
prompt = CustomOpenAIFunctionsAgent.create_prompt(system_message=SystemMessage(content=prefix))
agent = AgentExecutor.from_agent_and_tools(
    agent=CustomOpenAIFunctionsAgent(llm=llm, prompt=prompt, tools=tools, verbose=True),
    tools=tools, verbose=True
)

[elokus]

It also works better when you add arg_schema Field to PythonAstREPLTool. Code:

class AstArgSchema(BaseModel):
    """A schema for the ast argument."""
    query: str = Field(description="A string formatted plain python script with imports and variables to execute.")

class PythonAstREPLTool(BaseTool):
    """A tool for running python code in a REPL."""

    name = "python_repl_ast"
    description = (
        "A Python shell. Use this to execute python commands. "
        "Input should be a valid python command. "
        "When using this tool, sometimes output is abbreviated - "
        "make sure it does not look abbreviated before using it in your answer."
    )
    globals: Optional[Dict] = Field(default_factory=dict)
    locals: Optional[Dict] = Field(default_factory=dict)
    sanitize_input: bool = True
    args_schema: Type[BaseModel] = AstArgSchema
    ...

[batmanscode]

hey @li-xiaohui

I had to initialize the agent manually, here is how.

dataset={"df": df}
tools = [CustomPythonAstREPLTool(locals=dataset)]
tool_names = [tool.name for tool in tools]
prompt = CustomOpenAIFunctionsAgent.create_prompt(system_message=SystemMessage(content=prefix))
agent = AgentExecutor.from_agent_and_tools(
    agent=CustomOpenAIFunctionsAgent(llm=llm, prompt=prompt, tools=tools, verbose=True),
    tools=tools, verbose=True
)

Very useful, thanks @falmanna

[batmanscode]

@falmanna, what are the relevant imports for CustomPythonAstREPLTool and CustomOpenAIFunctionsAgent?

[falmanna]

@falmanna, what are the relevant imports for CustomPythonAstREPLTool and CustomOpenAIFunctionsAgent?

from langchain.callbacks import StdOutCallbackHandler
from langchain.schema import LLMResult
from langchain.tools.python.tool import PythonAstREPLTool
from pandasql import sqldf
from langchain.agents import Tool
from langchain.agents.openai_functions_agent.base import (
    OpenAIFunctionsAgent, 
    _format_intermediate_steps, 
    _FunctionsAgentAction
)
from langchain.schema import (
    AgentAction,
    AgentFinish,
    AIMessage,
    BaseMessage,
)
from langchain.callbacks.manager import Callbacks
from typing import Any, List, Tuple, Union, Dict
from json import JSONDecodeError

[Saksham2703]

hey @li-xiaohui I had to initialize the agent manually, here is how.

dataset={"df": df}
tools = [CustomPythonAstREPLTool(locals=dataset)]
tool_names = [tool.name for tool in tools]
prompt = CustomOpenAIFunctionsAgent.create_prompt(system_message=SystemMessage(content=prefix))
agent = AgentExecutor.from_agent_and_tools(
    agent=CustomOpenAIFunctionsAgent(llm=llm, prompt=prompt, tools=tools, verbose=True),
    tools=tools, verbose=True
)

Very useful, thanks @falmanna

Hey What is the SystemMessaage and how do you import it?

[falmanna]

Hey What is the SystemMessaage and how do you import it?

from langchain.agents import initialize_agent
from langchain.agents import AgentType
from langchain.agents import create_pandas_dataframe_agent
from langchain.agents.chat.base import ChatAgent
from langchain.agents.openai_functions_agent.base import OpenAIFunctionsAgent
from langchain.tools.python.tool import PythonAstREPLTool
from langchain.agents.chat.output_parser import ChatOutputParser
from langchain.agents.agent import AgentExecutor
from langchain.callbacks import StdOutCallbackHandler
from langchain.callbacks.base import BaseCallbackManager
from langchain.schema import SystemMessage

[snehilms]

hey @li-xiaohui

I had to initialize the agent manually, here is how.

dataset={"df": df}
tools = [CustomPythonAstREPLTool(locals=dataset)]
tool_names = [tool.name for tool in tools]
prompt = CustomOpenAIFunctionsAgent.create_prompt(system_message=SystemMessage(content=prefix))
agent = AgentExecutor.from_agent_and_tools(
    agent=CustomOpenAIFunctionsAgent(llm=llm, prompt=prompt, tools=tools, verbose=True),
    tools=tools, verbose=True
)

Where are you using the create_pandas_dataframe agent? I want to use this agent with gpt-3.5 model with OPENAIFUNCTION and was getting the same error, after implementing your solution the custom agent was not able to access the dataframes it seems. what agent are you using is it your custom agent or the pandas dataframe agent

[falmanna]

Where are you using the create_pandas_dataframe agent? I want to use this agent with gpt-3.5 model with OPENAIFUNCTION and was getting the same error, after implementing your solution the custom agent was not able to access the dataframes it seems. what agent are you using is it your custom agent or the pandas dataframe agent

This is basically creating a custom pandas dataframe. Use it as is instead of create_pandas_dataframe , pass in any LLM you like.

[snehilms]

Where are you using the create_pandas_dataframe agent? I want to use this agent with gpt-3.5 model with OPENAIFUNCTION and was getting the same error, after implementing your solution the custom agent was not able to access the dataframes it seems. what agent are you using is it your custom agent or the pandas dataframe agent

This is basically creating a custom pandas dataframe. Use it as is instead of create_pandas_dataframe , pass in any LLM you like.

It cannot find the tool I have CSV files which I am using but it cannot find the tools python

[snehilms]

uniswap_daily_interest = pd.read_csv("uniswap_daily_interest.csv")

dataset={"df": uniswap_daily_interest} tools = [CustomPythonAstREPLTool(locals=dataset)] tool_names = [tool.name for tool in tools] prompt = CustomOpenAIFunctionsAgent.create_prompt(system_message=SystemMessage(content="You are financial Analyst and answer all question based on the data provided")) agent = AgentExecutor.from_agent_and_tools( agent=CustomOpenAIFunctionsAgent(llm=ChatOpenAI(temperature=0,model="gpt-3.5-turbo"), prompt=prompt, tools=tools, verbose=True), tools=tools, verbose=True )

I have provided the dataset but still it output the error saying "no dataset has been provided please provide a dataset"

[falmanna]

uniswap_daily_interest = pd.read_csv("uniswap_daily_interest.csv")

dataset={"df": uniswap_daily_interest} tools = [CustomPythonAstREPLTool(locals=dataset)] tool_names = [tool.name for tool in tools] prompt = CustomOpenAIFunctionsAgent.create_prompt(system_message=SystemMessage(content="You are financial Analyst and answer all question based on the data provided")) agent = AgentExecutor.from_agent_and_tools( agent=CustomOpenAIFunctionsAgent(llm=ChatOpenAI(temperature=0,model="gpt-3.5-turbo"), prompt=prompt, tools=tools, verbose=True), tools=tools, verbose=True )

I have provided the dataset but still it output the error saying "no dataset has been provided please provide a dataset"

You need to tell the LLM what are the datasets variables names in the system message.

The LLM doesn't know what is inside the python env, so you have to tell it.

[xiaofeifei218]

I also encountered a similar problem (when using gpt-3.5-turbo), but this issue disappeared when I switched to using gpt4.

[SavvasMohito]

I also encountered a similar problem (when using gpt-3.5-turbo), but this issue disappeared when I switched to using gpt4.

+1 on that, I can verify that gpt-4 respects the given instructions on how to format the final answer. it seems to be an llm issue rather than a langchain agent issue

[lbolanos]

This is my solution for langchain 0.0.276:

from langchain.agents.agent_toolkits.pandas.base import _get_functions_prompt_and_tools class AstArgSchema(BaseModel): """A schema for the ast argument.""" query: str = Field(description="A string formatted plain python script with imports and variables to execute.")

prompt, tools = _get_functions_prompt_and_tools( df, ) tools[0].args_schema = AstArgSchema
agent_kwargs = { 'system_message': prompt.messages[0] } agent = initialize_agent( tools, llm, agent=AgentType.OPENAI_FUNCTIONS, agent_kwargs=agent_kwargs, verbose=True)

[AtheerAlgherairy]

@falmanna

I was trying to use your code for CustomPythonAstREPLTool and CustomOpenAIFunctionsAgent but I got the following import error.. any ideas to fix it?

[falmanna]

@falmanna

I was trying to use your code for CustomPythonAstREPLTool and CustomOpenAIFunctionsAgent but I got the following import error.. any ideas to fix it?

This solution was for a very old version, something around 0.200 Could be the problem.

[prova]

This is still happening though with gpt-3.5-turbo.

[MegalaRamu]

Still facing this issue. Anyone resolved this recently