Observations and Limitations of API Tools

[djpecot]

Wanted to leave some observations similar to the chain of thought smirk that was mentioned over here for JSON agents.

You can copy this snippet to test for yourself (Mine is in Colab atm)

# Install dependencies
!pip install huggingface_hub cohere --quiet
!pip install openai==0.27.0 --quiet
!pip install langchain==0.0.107 --quiet

# Initialize any api keys that are needed
os.environ["OPENAI_API_KEY"] = ""
os.environ["HUGGINGFACEHUB_API_TOKEN"] = ""

import os
from langchain import LLMChain, OpenAI, Cohere, HuggingFaceHub, Prompt, SQLDatabase, SQLDatabaseChain
from langchain.model_laboratory import ModelLaboratory
from langchain.agents import create_csv_agent, load_tools, initialize_agent, Tool
from langchain import LLMMathChain, OpenAI, SerpAPIWrapper, SQLDatabase, SQLDatabaseChain
import requests
from langchain.agents import create_openapi_agent
from langchain.agents.agent_toolkits import OpenAPIToolkit
from langchain.llms.openai import OpenAI
from langchain.requests import RequestsWrapper
from langchain.tools.json.tool import JsonSpec
import yaml
import urllib
# Core LLM model
llm = OpenAI(temperature=0)

# Create an API agent

# LoveCraft - .yaml API
yaml_url = "https://raw.githubusercontent.com/APIs-guru/openapi-directory/main/APIs/randomlovecraft.com/1.0/openapi.yaml"
urllib.request.urlretrieve(yaml_url, "lovecraft.yml")
with open("/content/lovecraft.yml") as f:
  data = yaml.load(f, Loader=yaml.FullLoader)
lovecraft_json_spec=JsonSpec(dict_=data, max_value_length=4000)
headers = {}
requests_wrapper=RequestsWrapper(headers=headers)
lovecraft_json_spec_toolkit = OpenAPIToolkit.from_llm(OpenAI(temperature=0), lovecraft_json_spec, requests_wrapper, verbose=True)
lovecraft_agent_executor = create_openapi_agent(
    llm=OpenAI(temperature=0),
    toolkit=lovecraft_json_spec_toolkit,
    verbose=True
)

tools = [
    # Tool(
    #     name = "Search",
    #     func=search.run,
    #     description="useful for when you need to answer questions about current events. You should ask targeted questions"
    # ),
    Tool(
        name="Lovecraft API - randomlovecraft.com",
         func=lovecraft_agent_executor.run,
         description="Access randomlovecraft.com documentation, process API responses, and perform GET and POST requests to randomlovecraft.com"
    )
]
mrkl = initialize_agent(tools, llm, agent="zero-shot-react-description", verbose=True)

mrkl.run("Can you get me a random Lovecraft sentence?")
mrkl.run("Can you get me a random Lovecraft sentence using /sentences`?")
mrkl.run("Using the `https://randomlovecraft.com/api` base URL, make a `GET` request random Lovecraft sentence using `/sentences`. Use a limit of 1.")

I pulled the openAPI example from the docs to do some testing on my own. I wanted to see if it was possible for an agent to fetch and process API docs and to be able to return the response for use in other applications. Some of the key noted things:

🐘Huge API Responses

If your response is large, it will directly affect the ability for the agent to ouput the response. This is because it goes into the token limit. Here the Observation is included in the token limit: a possible workaround for this could be routing the assembled requests over to a server / glue tool like make. Still haven't looked at other tools to see what's possible! Perhaps just wrapping that request in another request to a server would suffice.

🤔Verbosity is Better?

There is a weird grey area between too detailed and not detailed enough when it comes to the user prompt: I.e. check these 3 prompts out:

✅Test1:

input:

Can you get me a random Lovecraft sentence

output:

> Finished chain.
A locked portfolio, bound in tanned human skin, held certain unknown and unnamable drawings which it was rumoured Goya had perpetrated but dared not acknowledge.

🙅‍♂️Test2:

input:

Can you get me a random Lovecraft sentence using /sentences?

output:

> Finished chain.
To get a random Lovecraft sentence, make a GET request to https://randomlovecraft.com/api/sentences?limit=1.

✅Test3:

input:

Using the `https://randomlovecraft.com/api` base URL, make a `GET` request random Lovecraft sentence using `/sentences`. Use a limit of 1.

output:

> Finished chain.
The sentence returned from the GET request to https://randomlovecraft.com/api/sentences?limit=1 is "Derby did not offer to relinquish the wheel, and I was glad of the speed with which Portsmouth and Newburyport flashed by."

🔑Expensive Key Search

I ran out of my free OpenAI credits in a matter of 5 hours of testing various tools with Langchain 😅. Be mindful of how expensive this is because each action uses a request to the core LLM. Specifically, I noticed that json_spec_list_keys is a simple layer-by-layer search where each nested object observation costs an LLM request. This can add up especially when looping. I guess workaround wise, you'd probably want to either use a lighter/cheaper model for the agent or have someway to "store" Action Inputs and Observations. I guess that can be solved here, haven't implemented yet.

All in all, this tool is absolutely incredible! So happy I stumbled across this library!

[djpecot]

@RizziGit93 I haven't looked back at this yet, been working on other things. Maybe this has been corrected? Will have to go back and test..

[dosubot[bot]]

Hi, @djpecot! I'm Dosu, and I'm here to help the LangChain team manage their backlog. I wanted to let you know that we are marking this issue as stale.

Based on my understanding of the issue, you raised some observations and limitations regarding the use of API tools in the LangChain library. Some of the key points mentioned include the token limit for large API responses, the need for a workaround when dealing with large responses, the challenge of finding the right level of verbosity in user prompts, and the cost of using OpenAI credits for testing. RizziGit93 also commented, expressing a similar situation and asking for a solution to the token limit problem. You responded, stating that you haven't looked into it yet but will test it again.

Before we close this issue, we wanted to check with you if it is still relevant to the latest version of the LangChain repository. If it is, please let us know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days.

Thank you for your contribution to the LangChain project, and please don't hesitate to reach out if you have any further questions or concerns!