brokk icon indicating copy to clipboard operation
brokk copied to clipboard

Anthropic tool calling

Open jbellis opened this issue 1 month ago • 1 comments

Anthropic (still) doesn't support tool_choice=any (i.e. required tool call) + thinking

This leads to a lot of wasted round trips as Brokk tries to get it to use a tool

Our options include

  1. Switch Anthropic models back to synthetic tool calls. This suboptimal because Anthropic models are trained to generate tool calls internally with an xml-ish format that requires less escaping, and are translated to json by another layer; forcing it to generate json directly loses this optimization.
  2. Extend our ToolContext to include a "default" option when no tool is called; this can only be a single String. So for Search kickoff it could be answer, but most tool call sites do not have a good option like this. (And we don't know yet if this will actually result in what we want, or just a lot of calls to "answer" when we really wanted a task list.)
  3. Dynamically switch to non-thinking mode when we need a tool call and don't get one so we only have to burn one extra request instead of an unbounded number.
  4. Default Anthropic models to thinking=off and warn users when they turn it on.

jbellis avatar Nov 26 '25 16:11 jbellis

I think we go with #4 and make sure our existing "provide a think tool to non-reasoning models" path works with "reasoning models with thinking disabled."

jbellis avatar Nov 27 '25 13:11 jbellis

Default Anthropic models to thinking=off

This is complete in 6dcddf3d8c851f784cd245497acff8d2efa90247.

make sure our existing "provide a think tool to non-reasoning models" path works with "reasoning models with thinking disabled."

FTR this does not work both because

  1. we only add "think" tool in tool-emulation mode and
  2. we don't check reasoning effort, only whether the model is inherently non-reasoning

That said: I'm going to leave this out for now because Anthropic has done a good job of training its models to "think out loud" in the text response when reasoning is disabled.

jbellis avatar Dec 15 '25 14:12 jbellis