It's not possible to detect which tool returned a result, if return_direct=True

[mariokostelac]

Since https://github.com/hwchase17/langchain/pull/997 the last action (with return_direct=True) is not contained in the response["intermediate_steps"] list.

That makes it impossible (or very hard and non-ergonomic at least) to detect which tool actually returned the result directly.

Is that design intentional?

[hwchase17]

not intentional - we should make it returned

[mariokostelac]

I took a look into it, but it seems that the structure changed significantly since then. If the tool is returning directly, it's wrapped in AgentFinish very early so even if we add it in the intermediate_steps in https://github.com/hwchase17/langchain/blob/5b0e747f9a82b27fd4dbba8c14ac9bab3ecb24c0/langchain/agents/agent.py#L512, we append AgentFinish, which already lost the information about which tool exactly created that output.

Option 1: preserve that information somewhere in FinishAgent object, but that object is not exposed to the library consumer (at least i'm not aware of it) so we'd need to find a way to do that.

Option 2 (better): move the wrapping into FinishAgent out of the original method and move it back into the loop of where intermediate steps are created.

There are other options too, but they all seem too intrusive.

Do you think option 2 would be acceptable?

[shaojun]

@mariokostelac i'm using this to detect if output was based from a tool:

def ask(self, input: str):
    output = self.agent_executor.invoke({"input": input})
    output_from_tool_name = None
    if output['intermediate_steps'] != None and len(output['intermediate_steps']) > 0:
        if output['intermediate_steps'][0] != None and len(output['intermediate_steps'][0]) > 0:
            if output['intermediate_steps'][0][0] != None:
                if output['intermediate_steps'][0][0].tool != None:
                    output_from_tool_name = output['intermediate_steps'][0][0].tool
                    print('tool: {} is selected for this output'.format(output_from_tool_name))
    if output_from_tool_name != None:
        return 'the output is from tool: {}, the output content is: {}...'.format(output_from_tool_name, output['output'])
    return output['output']

but have no way to know if the output is directly return_direct from a tool and then send to me, or from tool output to llm, then to me.

and I noticed one thing that could be a bug? if the tool (already set return_direct to True) output is quite long, like here I was parsing text from a pdf, the parsed text string length is 117083, the memory in agent will be cleared.