Usage of `return_intermediate_step` on Agents, and agent step callbacks

[tbscode]

I was wondering how to use the return_intermediate_step flag for agent executors this is the current Appoach I found to be working:

Ok ive dug a little deeper and it seems like setting return_intermediate_step = True when creating the agent with initialize_agent works. Only when using a memory you need to set memory.output = "output" otherwise it will error when trying to save the context.

I had to do a minor modification in https://github.com/hwchase17/langchain/blob/894c272a562471aadc1eb48e4a2992923533dea0/langchain/memory/chat_memory.py#L32-L36

Cause when using agents the outputs can be lists wich would error when saving the context. If I modify it like this:

    def save_context(self, inputs: Dict[str, Any], outputs: Dict[str, str]) -> None:
        """Save context from this conversation to buffer."""
        input_str, output_str = self._get_input_output(inputs, outputs)
        if not isinstance(input_str, list):
            input_str = [input_str]
        if not isinstance(output_str, list):
            output_str = [output_str]

        for input in input_str:
            self.chat_memory.add_user_message(input)
        for output in output_str:
            self.chat_memory.add_ai_message(output)

Then even saving the context with memory works for the agent. ( you can also load an inital context from dict.

import os
from langchain.callbacks import get_openai_callback
from langchain.agents import Tool
from langchain.agents import AgentType
from langchain.memory import ConversationBufferMemory
from langchain import OpenAI
from langchain.utilities import GoogleSearchAPIWrapper
from langchain.schema import messages_from_dict
from langchain.agents import initialize_agent
from langchain.chat_models import ChatOpenAI
from langchain.callbacks.base import CallbackManager
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
from langchain.memory import ConversationBufferMemory, ConversationSummaryBufferMemory, ConversationSummaryMemory

from langchain import OpenAI, ConversationChain
from langchain.prompts.chat import (
    ChatPromptTemplate,
    SystemMessagePromptTemplate,
    AIMessagePromptTemplate,
    HumanMessagePromptTemplate,
)
from langchain.prompts import (
    ChatPromptTemplate,
    MessagesPlaceholder,
    SystemMessagePromptTemplate,
    HumanMessagePromptTemplate
)

llm = ChatOpenAI(temperature=0, openai_api_key=OPENAI_KEY, model_name="gpt-3.5-turbo",
                 streaming=True, callback_manager=CallbackManager([StreamingStdOutCallbackHandler()]))

memory = ConversationSummaryBufferMemory(
    return_messages=True, llm=llm, max_token_limit=150, memory_key="chat_history")

# set the output key so that memory doesn't error on save
memory.output_key = "output"

# You an input a previously saved agent state. like this:
state = [{'type': 'ai', 'data': {
    'content': 'Nice to meet you, Tim!', 'additional_kwargs': {}}}]
search = GoogleSearchAPIWrapper(
    google_api_key=GOOGLE_API_KEY, google_cse_id=my_cse_id)
tools = [
    Tool(
        name="Current Search",
        func=search.run,
        description="useful for when you need to answer questions about current events or the current state of the world"
    ),
]

memory.chat_memory.messages = messages_from_dict(state)

agent_chain = initialize_agent(
    tools, llm, agent=AgentType.CONVERSATIONAL_REACT_DESCRIPTION, verbose=True, memory=memory, return_intermediate_steps=True)

MESSAGE = "What is currently the most popular web browser?"

with get_openai_callback() as cb:

    out = agent_chain(
        inputs=[MESSAGE])


# The output dict will also contain a 'intermediate_steps' key.

example output:

out = {
...,
  "intermediate_steps: [(AgentAction(tool='Current Search', tool_input='current most popular web browser', log='Do I need to use a tool? Yes\nAction: Current Search\nAction Input: current most popular web browser'), "Zooming into the internet browser market shares on different platforms, Chrome continues to dominate as the most popular browser on desktops with a market share\xa0... Feb 21, 2023 ... The most popular current browsers are Google Chrome, Apple's Safari, Microsoft Edge, and Firefox. Historically one of the large players in the\xa0... The usage share of web browsers is the portion, often expressed as a percentage, of visitors to a group of web sites that use a particular web browser. This graph shows the market share of browsers worldwide from Mar 2022 - Mar ... 56% 70% Chrome Safari Edge Firefox Samsung Internet Opera UC Browser Android\xa0... Feb 11, 2021 ... Google Chrome, then, is by far the most used browser, accounting for more than half of all web traffic, followed by Safari in a distant second\xa0... Here we examine the top five browsers in the US, in order of popularity. ... basically a pinned tab of recent sites that syncs between the desktop and\xa0... Google Chrome is the most popular and widely-used desktop web browser by far. ... This browser's current market share is slightly less than it was at this\xa0... Mar 15, 2023 ... Firefox; Google Chrome; Microsoft Edge; Apple Safari; Opera; Brave; Vivaldi; DuckDuckgo; Chromium; Epic. Comparison of Best Browser\xa0... Web browser, cookie & cache settings. HealthCare.gov is compatible with most popular web browsing software. This includes the most recent and commonly used\xa0... Browser Statistics. ❮ Home Next ❯. W3Schools' famous ... The Most Popular Browsers ... W3Schools' statistics may not be relevant to your web site.")]
...

This is most definitely not the right way to do this but also I'm not sure if there is a correct way yet :D

What I think would be really cool is to have something like a callback_manager also for agent actions. That way you could develop applications using agents with immediate feedback while the agent is executed.

[tbscode]

Ok on second look callback manager does implement the agent callbacks already:

Using

  def new_agent_action(*args, **kwargs):
      print("AGENT ACTION", args, kwargs, flush=True)

  with get_openai_callback() as cb:
      cb.on_llm_new_token = new_llm_token
      cb.on_agent_action = new_agent_action

      out = agent_chain(
          inputs=[MESSAGE]

I do get the callbacks for actions taken by an agent. Hope this helps somebody :D

[tbscode]

This does also work without setting return_intermediate_step = True thus not requiring the workaround with saving memory context!

[SlapDrone]

Hey @tbscode! Trying to solve this problem myself (hooking into intermediate agent steps). Would you possibly be able to elaborate a bit on how you did this? Here's what I'm trying (based on what you've put above), but it isn't working as expected.

from langchain.callbacks import get_openai_callback

# ...
agent_chain.return_intermediate_steps=True
msg = "How does tinyvit work?"

def new_agent_action(*args, **kwargs):
    print("AGENT ACTION", args, kwargs, flush=True)

with get_openai_callback() as cb:
    #cb.on_llm_new_token = new_llm_token # not sure I understand this?
    cb.on_agent_action = new_agent_action

    out = agent_chain(
      inputs=[msg]
    )

I get:

AGENT ACTION (AgentAction(tool='Vector Index (TinyViT GitHub)', tool_input='How does tinyvit work?', log='\nThought: Do I need to use a tool? Yes\nAction: Vector Index (TinyViT GitHub)\nAction Input: How does tinyvit work?'),) {'run_id': UUID('2c78766d-7245-4a3a-9c08-34a7fb997a83'), 'parent_run_id': None, 'color': 'green'}
[...]
ValueError: One output key expected, got dict_keys(['output', 'intermediate_steps'])

Thanks!

[dosubot[bot]]

Hi, @tbscode! I'm Dosu, and I'm helping the LangChain team manage their backlog. I wanted to let you know that we are marking this issue as stale.

From what I understand, the issue is about using the return_intermediate_step flag for agent executors. You found a working approach by setting return_intermediate_step = True when creating the agent with initialize_agent, but had to make a minor modification in the code to handle lists as outputs. However, in the comments, you discovered that the callback manager already implements agent callbacks and that setting return_intermediate_step = True is not required, eliminating the need for the workaround. You also provided clarification on how to implement the callback manager to another user.

Before we close this issue, we wanted to check if it is still relevant to the latest version of the LangChain repository. If it is, please let us know by commenting on the issue. Otherwise, feel free to close the issue yourself or it will be automatically closed in 7 days.

Thank you for your contribution to LangChain! Let us know if you have any further questions or concerns.