feat: gpt-5 free-form tool calling support

[priyansh4320]

Why are these changes needed?

GPT‑5 can now send raw text payloads - anything from Python scripts to SQL queries - to your custom tool without wrapping the data in JSON using the new tool "type": "custom".

Related issue number

closes #2010

Checks

llm_config = LLMConfig(
    model="gpt-5-mini",
    api_key=os.getenv("OPENAI_API_KEY"),
    api_type="openai",
)

assistant = AssistantAgent("assistant", llm_config=llm_config)
user_proxy = UserProxyAgent(
    "user_proxy",
    code_execution_config={"work_dir": "coding", "use_docker": False},
    is_termination_msg=lambda x: x.get("content", "").startswith("de_something"),
)

@assistant.register_for_llm(free_form=True)
@user_proxy.register_for_execution()
def do_something(message: str):
    """
    This is a tool that performs a specific function.
    Args:
        message: The message to do something.
    Returns:
        The result of doing something.
    """
    return "do_something"


user_proxy.initiate_chat(assistant, message="use tool do something")

import os
from autogen.agentchat.group.patterns.auto import AutoPattern
from autogen.agentchat import run_group_chat
from autogen.io.run_response import Cost
from autogen import LLMConfig, ConversableAgent
from dotenv import load_dotenv
load_dotenv()


llm_config = LLMConfig(
    model="gpt-5-mini",
    api_key=os.getenv("OPENAI_API_KEY"),
    api_type="openai",
)


with llm_config:
    triage_agent = ConversableAgent(
        name="triage_agent",
        system_message="""You are a triage agent. For each user query,
        identify whether it is a technical issue or a general question. Route
        technical issues to the tech agent and general questions to the general agent.
        Do not provide suggestions or answers, only route the query.""",
        llm_config=llm_config
    )

    tech_agent = ConversableAgent(
        name="tech_agent",
        system_message="""You solve technical problems like software bugs
        and hardware issues.""",
        llm_config=llm_config
    )

    general_agent = ConversableAgent(
        name="general_agent", system_message="You handle general, non-technical support questions.",llm_config=llm_config
    )

user = ConversableAgent(name="user", human_input_mode="ALWAYS")

pattern = AutoPattern(
    initial_agent=triage_agent,
    agents=[triage_agent, tech_agent, general_agent],
    user_agent=user,
    group_manager_args={"llm_config": llm_config},
)


@triage_agent.register_for_llm(description="use this tool to triage the user query and return the output",free_form=True)
@user.register_for_execution()
def triage_agent_func(triage_output: str) -> str:
    return triage_output

response = run_group_chat(
    pattern=pattern, messages="My laptop keeps shutting down randomly. Can you help?", max_rounds=15
).process()

• [ ] I've included any doc changes needed for https://docs.ag2.ai/. See https://docs.ag2.ai/latest/docs/contributor-guide/documentation/ to build and test documentation locally.
• [ ] I've added tests (if relevant) corresponding to the changes introduced in this PR.
• [ ] I've made sure all auto checks have passed.

[priyansh4320]

@Lancetnik updated changes to branch

[marklysze]

@priyansh4320 thanks for this, a few questions:

• What's the difference between a tool with a string parameter and this?
• What happens when you run this with a non-GPT5 OpenAI model (e.g. gpt-4o-mini)
• What happens when you run this with a non-OpenAI model (e.g. Gemini Pro)
• Does this work if you add the tool to an agent through the agent's functions parameter?
• Can you help explain the meaning of the new role="Tool"
• This isn't that clear to me free_form (bool): Whether to allow the tool to be a free-form string. in terms of what free-form string means. Is the tool itself a string, or takes in a string.
• Does this work with streaming?

Some updates:

• Can you please add docstrings to ToolCall in agent_events.py to describe the new parameters
• Can we please always use run or run_group_chat - that's preferred over initiate_chat

[priyansh4320]

@marklysze ,

1. The difference is that GPT‑5 can now send raw text payloads to your custom tool using the new tool "type": "custom" and will allow us to integrate a new feature called context-free grammar (CFG) where we can enable users to define Lark, regex grammar rules in the tool spec.
2. This feature is supported by GPT-5 and above.
3. Not supported by non-OpenAI models
4. Works with decoration register_with_llm, tool and register_function() method as of now.
5. Not a new role 'tool', it is a correction I noticed in the docstring.
6. fixed the docstrings.
7. Streaming is not supported as of now.

[Lancetnik]

@randombet I don't see a chance to implement it another way in the current codebase. We need to create a lot of interfaces and our own DTO's to make such things scalable. Until this moment we can just work with raw dictionaries and make the things like this. I suggest to revisit this place in the future as a part of bigger refactoring job.

[randombet]

Discussed with @priyansh4320 offline and he is onboard. We could use this case to investigate the limitation of conversable agent and clients on how we could support model specific features without limit the generosity of conversable agent.

[codecov[bot]]

Codecov Report

:x: Patch coverage is 50.00000% with 23 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
autogen/agentchat/conversable_agent.py	50.00%	8 Missing and 4 partials :warning:
autogen/events/agent_events.py	57.14%	5 Missing and 1 partial :warning:
autogen/tools/tool.py	37.50%	4 Missing and 1 partial :warning:

Files with missing lines	Coverage Δ
autogen/tools/tool.py	85.93% <37.50%> (-7.17%)	:arrow_down:
autogen/events/agent_events.py	95.99% <57.14%> (-1.65%)	:arrow_down:
autogen/agentchat/conversable_agent.py	62.51% <50.00%> (-2.11%)	:arrow_down:

... and 28 files with indirect coverage changes

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.