ag2 [Issue]: Revisit LLMConfig and LLMConfigEntry Design

Describe the issue

Which parameters should reside in LLMConfig and LLMConfigEntry? Why use pydantic extra forbidden in LLMConfig and LLMConfigEntry?

Steps to reproduce

No response

Screenshots and logs

No response

Additional Information

No response

Jul 11 '25 23:07 randombet

Proposal: The output schema should be moved out of LLMConfig and made a part of generate_reply or initiate_chat group of functions.

Rationale: This isn't really a LLM Config. This is a use case config. For the same LLM Config (model name, temperature etc), I want the flexibility to define multiple response formats depending on my use case.

Jul 17 '25 13:07 r4881t

@randombet @r4881t @sonichi @qingyun-wu @marklysze @CAROLZXYZXY

Reviewed the LLMConfig. I have a few refactoring proposals.

'Tools' field to be moved out of LLMConfig.
'Functions' field to be moved out of LLMConfig.
Agree with @r4881t to move out 'response_format' from LLMConfig.

Rationale:

Tools, Functions, and Response_format are agent-specific fields and not LLM-specific.
The LLM-Specific configs are.

                temperature: Optional[float] = None
                check_every_ms: Optional[int] = None
                max_new_tokens: Optional[int] = None
                seed: Optional[int] = None
                allow_format_str_template: Optional[bool] = None
                response_format: Optional[Union[str, dict[str, Any], BaseModel, Type[BaseModel]]] = None
                timeout: Optional[int] = None
                cache_seed: Optional[int] = None
                routing_method: Optional[Literal["fixed_order", "round_robin"]] = None

& config_list

Since Tools and Functions are agent-specific fields, these should be handled as kwargs, since we are creating signatures for functions and tools and are using llm_config to register_with_llm.
Moving the response_format out of the LLMConfig will decentralise the State Transfer between agents and add more flexibility for state modifications during GroupChats. This will be an enhancement. and will allow us to build complex flows.

Jul 21 '25 20:07 priyansh4320

@priyansh4320 The rational to revisit this is the current LLMConfig and LLMConfigEntry is not generic enough to support all OAI compatible models and we can't support each model one by one.

To test how the new design works, I suggest find a new openai compatible model (e.g. grok) to see how the new design applies to Grok-4. If it support all Grok-4 features with the default OpenAI client and grok-4 config parameters.

Also need to check if this break the current groupchat, two agent chant (initiate_chat method) and single agent run method.

Jul 22 '25 19:07 randombet

I think, we could take inspiration from another tools:

PydanticAI: – Settings: https://github.com/pydantic/pydantic-ai/blob/main/pydantic_ai_slim/pydantic_ai/settings.py – Agent class: https://github.com/pydantic/pydantic-ai/blob/main/pydantic_ai_slim/pydantic_ai/agent.py#L262-L287

Smallagents: – Agent: https://github.com/huggingface/smolagents/blob/main/src/smolagents/agents.py#L259-L278 – Settings class tree: https://github.com/huggingface/smolagents/blob/main/src/smolagents/models.py#L1036

Continue Plugin config: https://docs.continue.dev/reference#models

https://github.com/Leezekun/MassGen

Aug 05 '25 20:08 Lancetnik

I think, we could move out any deprecated option from LLMConfig and save compatibility the way like this:

class LLMConfig:
     def __init__(self, tools: Sequence[Tool] = ()) -> None:
          if tools:
               warnings.warn(..., category=DeprecationWarning)

          self.tools = tools

class Agent:
     def __init__(self, config: LLMConfig, tools: Sequence[Tool] = ()) -> None:
          self.config = config
          self.tools = (*tools, *config.tools)

In this case, we could save the old API for a while, add a proper deprecation notice, and migrate to the new API smoothly.

Aug 06 '25 05:08 Lancetnik

Well, it seems like I have found some problems and have an idea of what the LLMConfig class should look like.

Firstly, we need to make it more typed

The user should be able to get IDE support working with our classes. They shouldn't have to guess about the options the LLMConfig requires or look up documentation on how to connect to the LLM. As a first step, I made a #2014 pull request to type the current LLMConfig API. However, the final version I suggest should look something like this:

ConfigItem = AnthropicLLMConfigTypedDict | BedrockLLMConfigTypedDict | ...

class LLMConfig:
    def __init__(self, *configs: ConfigItem | LLMConfigEntry | dict[str, Any]) -> None:
        ...

This way our users get a full typing support for LLMConfig's keys and types and the usage should looks like

# single config
LLMConfig({"model": "gpt-5", "api_token": ...})

# miltiple configs
LLMConfig({"model": "gpt-5", "api_token": ...}, {"model": "gpt-o3", "api_token": ...})

Features to deprecate and remove

So, current LLMConfig(model="gpt-5") and LLMConfig(config_list=[...]) / LLMConfig(config_list={"model": "gpt-5"}) API will be deprecated and scheduled to remove in 0.10.0.

Also, I suggest to deprecate and remove LLMConfig contextmanager, current, default and dict-compatible API. And, of course, it shouldn't load any API keys from environment variables by itself. Dicussed here – https://github.com/ag2ai/ag2/issues/1412#issuecomment-3168371951

LLMConfig should contains typed constructor and where feature - only. I still not sure about dumps and loads (I think, user could be able to make it by himself if it is required), but the main goal – make LLMConfig stongly typed, predictable and easy-to-support.

Extra options

As I can see, extra options are forbidden currently to catch users typos: pydantic will raise an error if users passes temperatura instead of temperature. But, I think, this problem could be solved by type hints. Thereby we can allow extra-options bypass after LLMConfig typing.

Providing extra options allows users to take advantage of new LLM features that we didn't have time to implement prior to public release, with correct typing.

Config tiers

Also, I see a 3 tiers of configs:

– Application-level – LLM-level – Agent-level

For sure, all of these tiers should has a level-specific configs, but we have a 2 options here:

Top-level configuration as a default

In this way, we can allow users to set application-level options as the default for nested levels. For example:

config = LLMConfig(  # Application level
    {  # LLM level
	    "model": "gpt-5",
	    "temperature": 0.5,  # LLM-level option has a bigger prior than application default
	    "api_token": ...
	}, { # Another LLM level
	    "model": "gpt-o3",
	    "api_token": ...
	},
	temperature=1  # will be used for o3 and other LLM without temperature
)

It is not so usefull for such options, but could be useful for tools and other mergeable options:

config = LLMConfig(
    {
	    "model": "gpt-5",
	    "tools": [tool2],
	    "api_token": ...
	}, {
	    "model": "gpt-o3",
	    "api_token": ...
	},
	tools=[tool1],
)

agent_5 = Agent(config.where(model="gpt-5"), tools=[tool3])
assert agent5.tools == [tool1, tool2, tool3]

agent_o3 = Agent(config.where(model="gpt-o3"), tools=[tool3])
assert agent_o3.tools == [tool1, tool3]

Do not mix config levels

In this case, we cannot merge or use default settings from the upper-level configuration, and the user must manually set them for each instance.

Therefore, a tool must be specified for each agent (for example).

config = LLMConfig(
    {
	    "model": "gpt-5",
	    "api_token": ...
	}, {
	    "model": "gpt-o3",
	    "api_token": ...
	},
)

agent_5 = Agent(config.where(model="gpt-5"), tools=[tool1])
assert agent5.tools == [tool1]

agent_o3 = Agent(config.where(model="gpt-o3"), tools=[tool1])
assert agent_o3.tools == [tool1]

It is a more strict and predictable approach, but it is also very rigid. I think we should prefer a more flexible approach and allow users to customize the application / LLM - levels settings.

Migration plan

[x] make current LLMConfig typed #2014
[x] make TypedDicts for all LLMConfigEntry #2019
[x] allow to use extra options to support untyped variants #2019
[x] add LLMConfig(*configs) API with LLMConfig(**kwargs) and LLMConfig(config_list=...) deprecation
[x] deprecate useless methods usage (LLMConfig.current, default, dict API, etc)
[x] update all documentation LLMConfig usages to remove deprecated syntax
[ ] make a decision about each option Config-level
[ ] add to Agent and other LLMConfig usage places related config-level options
[ ] write a lot of tests 😄