Agent doesn't always select custom tool, even with `tool_choice="required"`

[anujsingla]

System Info

:128: RuntimeWarning: 'torch.utils.collect_env' found in sys.modules after import of package 'torch.utils', but prior to execution of 'torch.utils.collect_env'; this may result in unpredictable behaviour Collecting environment information... PyTorch version: 2.6.0 Is debug build: False CUDA used to build PyTorch: None ROCM used to build PyTorch: N/A

OS: macOS 15.3.1 (arm64) GCC version: Could not collect Clang version: 16.0.0 (clang-1600.0.26.6) CMake version: version 3.30.5 Libc version: N/A

Python version: 3.12.7 | packaged by Anaconda, Inc. | (main, Oct 4 2024, 08:22:19) [Clang 14.0.6 ] (64-bit runtime) Python platform: macOS-15.3.1-arm64-i386-64bit Is CUDA available: False CUDA runtime version: No CUDA CUDA_MODULE_LOADING set to: N/A GPU models and configuration: No CUDA Nvidia driver version: No CUDA cuDNN version: No CUDA HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True

CPU: Apple M1 Pro

Versions of relevant libraries: [pip3] flake8==7.0.0 [pip3] mypy==1.11.2 [pip3] mypy-extensions==1.0.0 [pip3] numpy==1.26.4 [pip3] numpydoc==1.7.0 [pip3] torch==2.6.0 [pip3] torchaudio==2.6.0 [pip3] torchvision==0.21.0 [conda] numpy 1.26.4 py312h7f4fdc5_0
[conda] numpy-base 1.26.4 py312he047099_0
[conda] numpydoc 1.7.0 py312hca03da5_0
[conda] torch 2.6.0 pypi_0 pypi [conda] torchaudio 2.6.0 pypi_0 pypi [conda] torchvision 0.21.0 pypi_0 pypi

Information

• [ ] The official example scripts
• [ ] My own modified scripts

🐛 Describe the bug

We've added one custom tool to our Llama Stack agent and set tool_choice="required" to make sure it always calls this tool. However, the agent only uses the tool correctly the first time we ask a question. After that, it ignores the tool and answers questions directly using the LLM.

Our Configuration

client_tools = [web_search]
agent_config = AgentConfig(
    model=selected_model,
    instructions=(
            "You are a AI chat assistant. Use the tools from the configuration you have access to for providing relevant answers.  You must always retrieve data from the backend system "
            "by calling the available tool, even if you think you already have the answer."
            "**Never assume an answer without making a tool call.**"
            "If a user asks for something, call the tool first before responding."
            "Do not assume an answer without using a tool."
        ),
    sampling_params={
        "strategy": {"type": "top_p", "temperature": 0.1, "top_p": 0.1},
    },
    toolgroups=[],
    client_tools=[
        client_tool.get_tool_definition() for client_tool in client_tools
    ],
    enable_session_persistence=True,
    tool_config={
        "tool_choice": "required",
        "tool_prompt_format": "python_list"
    },
)
agent = Agent(client, agent_config, client_tools)

Actual Behavior

It calls the tool the first time, but afterward, it ignores the tool and responds directly from the LLM.

How to Reproduce

1. Create an agent with one custom tool.
2. Set tool_choice="required".
3. Ask the agent multiple questions.
4. Notice it doesn't call the tool after the first question.

Environment

• Llama Stack version: 0.1.5
• Model: llama3.2:3b-instruct-fp16

Can you please check why this happens or suggest how to ensure the tool is always called?

Error logs

N/A

Expected behavior

The agent should always call our custom tool before responding, every single time.