ml-commons
ml-commons copied to clipboard
opensearch-project
[BUG] Conversational agent overwrites input from SearchIndexTool
It is straightforward and works of course when I define a SearchIndexTool inside a conversational flow agent, the question is inserted into the defined OpenSearch query, there will come search results.
Unfortunately, I can not manage this with a conversational agent.
This agent should always execute the same query:
{
"name": "Test Agent",
"type": "conversational",
"description": "Simple agent to test the agent framework",
"llm": {
"model_id": "{{ _.LlmModelId4agent }}",
"parameters": {
"max_iteration": 5,
"stop_when_no_tool_found": true,
"disable_trace": false
}
},
"memory": {
"type": "conversation_index"
},
"app_type": "chat_with_rag",
"tools": [
{
"type": "SearchIndexTool",
"description": "A tool to search opensearch index with natural language question. If you don't know answer for some question, you should always try to search data with this tool. Action Input: <natural language question>",
"include_output_in_agent_response": true,
"parameters": {
"input": "{\"index\": \"scll_agendaitem\", \"query\": {\"query\": { \"match_all\": {}}} }"
}
}
]
}
I get always this error message: Failed to run the tool SearchIndexTool with the error message Expected a com.google.gson.JsonObject but was com.google.gson.JsonPrimitive; at path $. Independent from the specified tool input. It seems that the conversational agent overwrites the parameters.input content with its generated, redefined query, wasting the query specification from the SearchIndexTool.
Expected behavior is that you define the OpenSearch query inside the tool description, and that the output from the former step of the conversational search should be inserted into this query, e.g. like this:
{
"type": "SearchIndexTool",
"description": "A tool to search opensearch index with natural language question. If you don't know answer for some question, you should always try to search data with this tool. Action Input: <natural language question>",
"parameters": {
"input": "{\"index\": \"${parameters.index}\", \"query\": ${parameters.query} }",
"query": {
"query": {
"match": {
"tns_body_chunked768": {
"query": "${parameters.FORMER_OUTPUT_OF_CONVERSATIONAL_AGENT}"
}
}
},
"size": "5",
"_source": "tns_body_chunked768"
}
}
}
Maybe the 'input' parameter for the query specification can be renamed or configured.
Hi @jngz-es ! Is this the same bug in https://github.com/opensearch-project/ml-commons/issues/2836? Will this PR fix: https://github.com/opensearch-project/ml-commons/pull/2837?
As far as I see, #2837 will fix it for MLConversationalFlowAgentRunner and MLFlowAgentRunner, but not for MLChatAgentRunner.
MLChatAgentRunner uses the AgentUtils method AgentUtils.constructToolParams for generating the params for a tool. This is done here. The PR fix creates a new method AgentUtils.getToolExecuteParams that will do this for MLConversationalFlowAgentRunner and MLFlowAgentRunner.
Hi @reuschling ! I guess you wish to somehow hardcode the search query for search index tool. Unfortunately, the tool input will be overridden by the conversational agent.
Since you are already hardcoding the search query, I recommend you to try flow agent to first fetch the results from search index tool. You can then concatenate a ml model tool or connector tool to call the large language model.
You can even specify your prompt in the ml model tool.
If it is a bug in flow agent and conversational flow agent, then it is also a bug inside conversational agent, aka MLChatAgentRunner. The cleanest way of fixing it would be also as part of #2837 . @jngz-es I can support if you want.
I doubt whether the same fix would work. In conversational agent, we let LLM to generate input parameters and pass them into the tools.
I will discuss with @jngz-es tomorrow to check whether this would introduce some breaking change.
I recommend you to try flow agent to first fetch the results from search index tool. You can then concatenate a ml model tool or connector tool to call the large language model.
@reuschling Could you try my workaround?
I want to use the conversational agent, but I want to have a tool (among others, e.g. one for hybrid search) which makes dedicated queries inside the index, such as counting documents for example. The conversational agent should give input for this, but this input should be inserted into a predefined query template, with an according parameter value reference inside. This is what the SearchIndexTool is made for. Using a flow agent instead is no solution for me.
The SearchIndexTool should behave same, independent if it is used inside flow or conversational agents. Maybe then the use of the input variable for the query template definition is wrong.
But I doubt it is not consistent or understandable if after merging #2837, all tools with some definitions inside the input variable will act differently dependent where they are used (flow or conversational). I.e. all tools (if there are others) with necessary input definitions currently don't work and won't work with conversational agent as these definitions will be overwritten. #2837 fixes this problem only for the flow agents.
But not overwriting the input if there is already an input definition inside the tool specification would solve this also for the conversational agent. This is what #2837 do.
Crucial is, that there is a possibility to get the formerly LLM generated action input inside a variable for referencing inside the query template. As ${parameters.question} in the tutorial example
We just have a discussion. The conclusion is that we would introduce a new field in tool settings like "configs", which works like hardcode parameters.
This sounds promising, cleaner as some hidden input prioritization. I'm looking forward to it, the current status is a blocker for me.
Cool, @reuschling thanks for reporting this issue. @jngz-es is working on this. Will publish PR soon
In my opinion, this bug is fixed with #2977 and #3200.
https://github.com/opensearch-project/ml-commons/pull/3200#issuecomment-2516579315 gives an example how to use a placeholder inside a tool used inside a conversational agent.
What is missing is the according documentation for the new config field possibility in tool specifications #2977 , where you can add a field 'input' to specify a static action input for the tool.
Inside this static action input you can use a placeholder where you can insert the origin action input created from the former agent instance:
${parameters.previous_tool_name.output}in the case the tool is used inside a flow agent${parameters.llm_generated_input}in the case the tool is used inside a conversational agent (works also with the new Plan-execute-reflect agent)
I commented that it is confusing that the same functionality has different placeholder names regarding where the tool is used, but it works.
Nevertheless, without regarding documentation this is hidden functionality for now.