Callbacks are ignored when passed to load_tools

[Lothiraldan]

Hello,

I cannot figure out how to pass callback when using load_tools, I used to pass a callback_manager but I understand that it's now deprecated. I was able to reproduce with the following snippet:

from langchain.agents import load_tools
from langchain.callbacks.base import BaseCallbackHandler
from langchain.tools import ShellTool


class MyCustomHandler(BaseCallbackHandler):
    def on_tool_start(self, serialized, input_str: str, **kwargs):
        """Run when tool starts running."""
        print("ON TOOL START!")

    def on_tool_end(self, output: str, **kwargs):
        """Run when tool ends running."""
        print("ON TOOL END!")


# load_tools doesn't works
print("LOAD TOOLS!")
tools = load_tools(["terminal"], callbacks=[MyCustomHandler()])
print(tools[0].run({"commands": ["echo 'Hello World!'", "time"]}))

# direct tool instantiation works
print("Direct tool")
shell_tool = ShellTool(callbacks=[MyCustomHandler()])
print(shell_tool.run({"commands": ["echo 'Hello World!'", "time"]}))

Here is the output I'm seeing:

LOAD TOOLS!
/home/lothiraldan/project/cometml/langchain/langchain/tools/shell/tool.py:33: UserWarning: The shell tool has no safeguards by default. Use at your own risk.
  warnings.warn(
Hello World!
user	0m0,00s
sys	0m0,00s

Direct tool
ON TOOL START!
ON TOOL END!
Hello World!
user	0m0,00s
sys	0m0,00s

In this example, when I pass the callbacks to load_tools, the on_tool_* methods are not called. But maybe it's not the correct way to pass callbacks to the load_tools helper.

I reproduced with Langchain master, specifically the following commit https://github.com/hwchase17/langchain/commit/a9c24503309e2e3eb800f335e0fbc7c22531bda0.

Pip list output:

Package                 Version   Editable project location
----------------------- --------- -------------------------------------------
aiohttp                 3.8.4
aiosignal               1.3.1
async-timeout           4.0.2
attrs                   23.1.0
certifi                 2022.12.7
charset-normalizer      3.1.0
dataclasses-json        0.5.7
frozenlist              1.3.3
greenlet                2.0.2
idna                    3.4
langchain               0.0.157   /home/lothiraldan/project/cometml/langchain
marshmallow             3.19.0
marshmallow-enum        1.5.1
multidict               6.0.4
mypy-extensions         1.0.0
numexpr                 2.8.4
numpy                   1.24.3
openai                  0.27.6
openapi-schema-pydantic 1.2.4
packaging               23.1
pip                     23.0.1
pydantic                1.10.7
PyYAML                  6.0
requests                2.29.0
setuptools              67.6.1
SQLAlchemy              2.0.12
tenacity                8.2.2
tqdm                    4.65.0
typing_extensions       4.5.0
typing-inspect          0.8.0
urllib3                 1.26.15
wheel                   0.40.0
yarl                    1.9.2

[PawelFaron]

@Lothiraldan Just give the callbacks to run:

print("LOAD TOOLS!")
tools = load_tools(["terminal"])
print(tools[0].run({"commands": ["echo 'Hello World!'", "time"]}, callbacks=[MyCustomHandler()]))

[Lothiraldan]

@PawelFaron Thanks for your suggestion but I think that will only works with that particular reproduction script. My actual use-case is similar to that example notebook: https://github.com/hwchase17/langchain/blob/master/docs/ecosystem/comet_tracking.ipynb Scenario 3: Using An Agent with Tools:

callbacks = [StdOutCallbackHandler(), comet_callback]
llm = OpenAI(temperature=0.9, callbacks=callbacks)

tools = load_tools(["serpapi", "llm-math"], llm=llm, callbacks=callbacks)
agent = initialize_agent(
    tools,
    llm,
    agent="zero-shot-react-description",
    callbacks=callbacks,
    verbose=True,
)

[PawelFaron]

@Lothiraldan Not sure if this can be a workaround for you:

print("LOAD TOOLS!")
tools = load_tools(["terminal"])
tools[0].callbacks = [MyCustomHandler()]
print(tools[0].run({"commands": ["echo 'Hello World!'", "time"]}))

The problem is that the shell tool belongs to base tools and it just loaded without any other parameters passing:

        elif name in _BASE_TOOLS:
            tools.append(_BASE_TOOLS[name]())

[PawelFaron]

For some other cases you can pass the CallbackManager as load_tools take it and use it in some casees

manager = CallbackManager([MyCustomHandler()])
tools = load_tools(["terminal"], callback_manager=manager)

[Lothiraldan]

I tested with langchain version 0.0.178 and it seems resolved, thank you very much!