Tool use fails with error 'Tool' object has no attribute 'name'

[kjw227]

Hi,

I'm trying to use the tools feature in guidance to track each time a grammar function is called in guidance (further down the line I would like to use this functionality to implement on-the-fly interpreters for programs generated by llms).

The bug

The problem is that tool calls are not working. A minimal example:

To reproduce

import guidance
from guidance import models, gen
from guidance import one_or_more, select, zero_or_more
from guidance import capture, Tool

MODELFILE = 'codellama/CodeLlama-7b-hf'

@guidance(stateless=True)
def number(lm):
  base = select(['0', '2', '5'])
  rec = number() + number()
  return lm + select([base, rec], name='tool_arg')

@guidance
def numbertool(lm):
  exp = lm['tool_arg']
  print('TOOLING: ' + str(exp))
  return lm

def get_model(model):
  return guidance.models.Transformers(model, device_map='auto')

if __name__ == '__main__':
  model = get_model(MODELFILE)
  prompt = 'How many states are there in the United States? '
  ntool = Tool(number(), numbertool)
  query_result = model + prompt + gen(max_tokens=10, tools=[ntool], stop = ' ')
  print(query_result)

What I expected to see is TOOLING: n logged on the console for each number generation (for example, if the llm generates 50, I would expect to see TOOLING: 5\nTOOLING: 0\nTOOLING: 50). Instead, I'm getting the following error trace:

Traceback (most recent call last):
  File "/home/ubuntu/llmtest/ihatellms/grammar_test.py", line 27, in <module>
    query_result = model + prompt + gen(max_tokens=10, tools=[ntool], stop = ' ')
  File "/home/ubuntu/.conda/envs/newllm/lib/python3.9/site-packages/guidance/models/_model.py", line 1212, in __add__
    out = value(lm)
  File "/home/ubuntu/.conda/envs/newllm/lib/python3.9/site-packages/guidance/_grammar.py", line 60, in __call__
    return self.f(model, *self.args, **self.kwargs)
  File "/home/ubuntu/.conda/envs/newllm/lib/python3.9/site-packages/guidance/library/_gen.py", line 147, in gen
    raise ValueError(f"Could not infer unambiguous tool call prefix for tool {tool.name}")
AttributeError: 'Tool' object has no attribute 'name'

The same behavior holds for more complex constraints that I've tried out. Interestingly though, the example from the readme works if I copy-paste it verbatim, but that's the only tool I've got working.

System info (please complete the following information):

• OS (e.g. Ubuntu, Windows 11, Mac OS, etc.): Ubuntu 22.04
• Guidance Version (guidance.__version__): 0.2.0

Thanks!

[nchammas]

Interestingly though, the example from the readme works if I copy-paste it verbatim, but that's the only tool I've got working.

Which example from the README are you referring to? Because there is an example that looks similar to yours (the one called calculator_call and calculator), and there is this other example with a nicer syntax:

https://github.com/guidance-ai/guidance/blob/546d9f037c8af739452939658ad6852192a64340/README.md#L52-L78

[kjw227]

The example that works is the one that uses calculator and calculator_call, later in the readme. Your linked example also works, except that it's giving me an error at the end (Error in monitoring: [Errno 32] Broken pipe).

I'm noticing that the example you linked is not calling the Tool constructor explicitly, whereas the calculator and calculator_call example does.

Actaully, the examples is not even assigning a grammar to match to the tool. This is surprising me: how is Guidance matching the add tool to occurrences of "add(x, y)" in the output of the LLM? The code doesn't even know that "add" is a function that takes parameters using parentheses!

[nchammas]

Actaully, the examples is not even assigning a grammar to match to the tool. This is surprising me: how is Guidance matching the add tool to occurrences of "add(x, y)" in the output of the LLM? The code doesn't even know that "add" is a function that takes parameters using parentheses!

I assume it "knows" because add is explicitly passed in to the tools parameter:

https://github.com/guidance-ai/guidance/blob/546d9f037c8af739452939658ad6852192a64340/README.md?plain=1#L77

[kjw227]

Yes, but doesn't the implementation of add not tell us anything about how add is used in what the llm generates?

 @guidance 
 def add(lm, input1, input2): 
     lm += f' = {int(input1) + int(input2)}' 
     return lm 

The function seems to tell us what to append to the llm, but not how to match cases of add generated by the llm. In this example, add is used as a prefix-style function name that takes arguments in parentheses, but what if add were an infix operator and the llm outputs something like 1 add 1 instead? Unless guidance is implicitly doing some mapping from the name of the tool-function itself?

[nchammas]

I don't know if infix notation is supported by Guidance. For example, I fixed your original code by adding an explicit prefix call to number() to show the model how to use it, plus a single training example:

import guidance
from guidance import gen, select, Tool


@guidance(stateless=True)
def number(lm):
  base = select(['0', '2', '5'])
  rec = number() + number()
  return lm + "number(" + select([base, rec], name='tool_args') + ")"


@guidance
def numbertool(lm):
  exp = lm['tool_args']
  print('TOOLING: ' + str(exp))
  return lm


if __name__ == '__main__':
    model = guidance.models.Transformers('codellama/CodeLlama-7b-hf', device_map='auto')
    model += "How many countries are there in the world? number(250) \n"
    ntool = Tool(number(), numbertool)
    model += 'How many states are there in the United States? '
    model += gen(max_tokens=10, tools=[ntool], stop=' ')
    print(model)

Output:

$ python test-tool.py 
Loading checkpoint shards: 100%|████████| 2/2 [00:13<00:00,  6.97s/it]
TOOLING: 5
How many countries are there in the world? number(250) 
How many states are there in the United States? number(5)

I also changed tool_arg to tools_args because of this comment that I noticed in the source for Tool, but that change doesn't seem necessary.

https://github.com/guidance-ai/guidance/blob/003917c5480a4b7cd4a447efbab40af5a223301a/guidance/library/_tool.py#L9

I wonder what the use is of this more verbose call_grammar + tool_call syntax vs. just passing the function to gen(tools=...). This could definitely use better documentation.