agents,tools: Proposal for more general interfaces of `tools` and `AgentAction `

[gluonfield]

Motivation

Currently langchaingo does not support multi argument tools. Standards for LLM tool calling in foundational models are emerging hence we should adapt the library to utilise models fully. In addition AgentAction should support multi argument tool input.

The goal of this PR is to

• [x] Review tool calling API's of several foundational model providers
• [ ] Reach consensus on langchaingo multi argument tool call interfaces
• [ ] Reach consensus on langchaingo agents tool calling interfaces
• [ ] Implement changes to support multi argument tool calls
• [ ] Implement changes to support agent multi argument tool calls

Review of Tool Call APIs

Most foundational models follows APIs which is this proposal fulfils.

OpenAI

Reference 1 Reference 2

Request

  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_current_weather",
        "description": "Get the current weather in a given location",
        "parameters": {
          "type": "object",
          "properties": {
            "location": {
              "type": "string",
              "description": "The city and state, e.g. San Francisco, CA"
            },
            "unit": {
              "type": "string",
              "enum": ["celsius", "fahrenheit"]
            }
          },
          "required": ["location"]
        }
      }
    }
  ],

Response

"tool_calls": [
        {
          "id": "call_FthC9qRpsL5kBpwwyw6c7j4k",
          "function": {
            "arguments": "{\"location\": \"San Francisco, CA\"}",
            "name": "get_rain_probability"
          },
          "type": "function"
        },
        {
          "id": "call_RpEDoB8O0FTL9JoKTuCVFOyR",
          "function": {
            "arguments": "{\"location\": \"San Francisco, CA\", \"unit\": \"Fahrenheit\"}",
            "name": "get_current_temperature"
          },
          "type": "function"
        }
      ]

Cohere

Reference 1 Reference 2

Identical to OpenAI up to field names and arguments is not string encoded.

Mistal

Reference 1

API is identical to OpenAI.

Google Vertex

Reference 1

Identical to OpenAI up to field names.

Proposal: updated Tool interface

type Tool interface {
	Name() string
	Description() string
	- Call(ctx context.Context, input string) (string, error)
	+ Call(ctx context.Context, input map[string]any) (string, error)
	+ Schema() Schema
}

type Schema struct {
	Type       string
	Properties map[string]any
	Required   []string
}

This is a breaking change replacing previous

Call(ctx context.Context, input string) (string, error)

and introducing Schema() method.

Example tool

type Tool struct {}

func (t Tool) Name() string {
	return "GoogleSearch"
}

func (t Tool) Description() string {
	return `
	"A wrapper around Google Search. "
	"Useful for when you need to answer questions about current events. "
	"Always one of the first options when you need to find information on internet"
	"Input should be a search query."`
}

func (t Tool) Call(ctx context.Context, input map[string]any) (string, error) {
	query, ok := input["query"].(string)
	if !ok {
		return "", fmt.Errorf("could not obtain `query` in %v", input)
	}
        // tool stuff
}

func (t Tool) Schema() tools.Schema {
	return tools.Schema{
		Type: "object",
		Properties: map[string]any{
			"query": map[string]any{
				"title": "query",
				"type":  "string",
			},
		},
		Required: []string{"query"},
	}
}

It's a job of the tool developer to define the schema and handling schema fields.

Proposal: updated AgentAction

For agents to support multi argument tools it's necessary to replace

type AgentAction struct {
	Tool      string
	ToolInput map[string]any
	Log       string
	ToolID    string
}

Which replaces ToolInput string. This allows straightforward propagation for tool arguments from LLM to AgentAction to Tool.

Let me know what you think @tmc @eliben. I'm happy to implement and handle breaking changes.

PR Checklist

• [x] Read the Contributing documentation.
• [ x Read the Code of conduct documentation.
• [x] Name your Pull Request title clearly, concisely, and prefixed with the name of the primarily affected package you changed according to Good commit messages (such as memory: add interfaces for X, Y or util: add whizzbang helpers).
• [x] Check that there isn't already a PR that solves the problem the same way to avoid creating a duplicate.
• [ ] Provide a description in this PR that addresses what the PR is solving, or reference the issue that it solves (e.g. Fixes #123).
• [ ] Describes the source of new concepts.
• [ ] References existing implementations as appropriate.
• [ ] Contains test coverage for new functions.
• [ ] Passes all golangci-lint checks.