Model Support

[jackgerrits]

Tracking issue for overall area of model support in AutoGen. See sub-issues for specific areas as well as following areas:

• We need to ensure that expectations are clear for which models are supported, and in what capacity.
• What model providers need support added?
• [Exploration] Automatic model selection
• [Exploration] Prompt fine tuning

[SongChiYoung]

Hi, I’d like to highlight a compatibility issue we encountered when trying to support the Mistral family.

Currently, some parts of the codebase use startswith() checks on model_family or model_name to determine model type. This worked fine in the past, but Mistral-style models use names like "mistral-large-latest" or "codestral-latest", which don’t match current assumptions.

This caused issues during integration, which we addressed in the following PR: #6158

To ensure robust support for models like Mistral, it would be helpful to audit and replace loose startswith() checks with a more reliable mapping or registry-based method (e.g., MODEL_POINTERS, etc.).

[nicsuzor]

Hi there, thanks so much for the recent updates. I just want to note an inconsistency between merge #6158 and the docs.

Specifically, at https://github.com/microsoft/autogen/commit/27da37efc0e14a8d2c193d732e6899895b14f7bd#r154885312, the change for Mistral etc introduces a new problem for those relying on the default. Previously, if a user provides an unregistered string for model_family, that would be interpreted as ModelFamily.UNKNOWN. Now it errors out on not finding a transformer instead. So the fix for 'codestral-latest' breaks e.g. 'llama-3-3-xyz', which in many cases would otherwise work perfectly fine with the default.

The docs suggest that autogen should provide a default for unknown model_family values:

Other families definitely exist and can be represented by a string, however, AutoGen will treat them as unknown

[SongChiYoung]

@nicsuzor Thanks for the feedback! I believe the current behavior still supports the default fallback logic.

In particular, ModelFamily.UNKNOWN is explicitly handled via:

for model in __unknown_models:
    register_transformer("openai", model, __BASE_TRANSFORMER_MAP)

And we also register:

register_transformer("openai", "default", __BASE_TRANSFORMER_MAP)

If you encountered a KeyError with "llama-3-3-xyz", could you share a minimal repro? It might be a mismatch with ModelFamily.ANY not including that string, or fallback lookup logic missing in get_transformer.

Happy to review this further if we can pinpoint the path where fallback fails!

And here is my testcase

client2 = OpenAIChatCompletionClient(
    model="llama3.1:latest",
    base_url="http://127.0.0.1:11434/v1",
    api_key="ollama",
    model_info={
        "vision": False,
        "function_calling": True,
        "json_output": True,
        "family": "unknown",
    },
)

messages = [
    UserMessage(content="hello", source="user"),
]

# print("OLLAMA SDK : ", asyncio.run(client1.create(messages=messages)))
print("OPENAI SDK : ", asyncio.run(client2.create(messages=messages)))
print(client2)
print("DONE IT")

Additional...

You're absolutely right that the fallback path in get_transformer() might look like it raises on unknown families:

if model_family == ModelFamily.UNKNOWN:
    model_family = _find_model_family(api, model)

But as you noted, _find_model_family() is itself designed to be fail-safe. Here’s the actual logic:

def _find_model_family(api: str, model: str) -> str:
    family = ModelFamily.UNKNOWN
    for _family in MESSAGE_TRANSFORMERS[api].keys():
        if model.startswith(_family):
            family = _family
    return family

So even if no match is found, it simply returns "unknown" and avoids crashing. That’s intentional – it’s meant to gracefully handle cases where a new model (e.g., "llama-3-3-xyz") isn’t explicitly mapped but might still inherit OpenAI-compatible behavior.

And yes, we use startswith for _find_model_family() because this is about unmapped models, and we want best-effort affinity. If there’s a better heuristic, I’m open to refining it. 😄

Appreciate your thoughtful review as always!

[SongChiYoung]

@nicsuzor Thanks again — I took a deeper dive and I believe I’ve now identified exactly what you were referring to. Just to confirm, is this the case you meant?

client2 = OpenAIChatCompletionClient(
    model="llama3.1:latest",
    base_url="http://127.0.0.1:11434/v1",
    api_key="ollama",
    model_info={
        "vision": False,
        "function_calling": True,
        "json_output": True,
        "family": "error",  # <--- arbitrary / invalid family
    },
)

If so, that totally makes sense.

I had assumed that by the time we enter the transformer layer, any unrecognized family would already be normalized to "unknown" — based on this docstring:

“Other families definitely exist and can be represented by a string, however, AutoGen will treat them as unknown.”

But I now realize that assumption was incorrect — there’s no explicit fallback normalization happening before get_transformer() is called. So yes, passing an arbitrary string (like "error") bypasses the safety net and leads to a missing transformer.

That gap is now clear to me, and I really appreciate you pointing it out.

To make the transformer behavior robust regardless of how model_info.family is supplied, I’ll submit a patch soon that ensures fallback to "unknown" if the family is invalid or unregistered at transformer logic .

Thanks again for surfacing this!

---

Now you should check solved that issue at #6213

[nicsuzor]

My apologies; yes, that's the error. Here's my test case:

from autogen_core.models import UserMessage
from autogen_ext.models.openai import (
    OpenAIChatCompletionClient,
)

model_cfg = {
    "api_key": "xyz",
    "base_url": "https://generativelanguage.googleapis.com/v1beta/openai/",
    "model_info": {
        "family": "gemini-2.5-pro",
        "function_calling": True,
        "structured_output": False,
        "json_output": True,
        "vision": False,
    },
    "model": "gemini-2.5-pro-exp-03-25",
}
client = OpenAIChatCompletionClient(**model_cfg)
messages = [UserMessage(content="What is the capital of France?", source="user")]
print("family ==  'gemini-2.5-pro':", end="")
try:
    response = await client.create(messages=messages)
    print(response.content)
except Exception as e:
    print(e)


print("family == 'gemini-1.5-pro':", end="")
model_cfg["model_info"]["family"] = "gemini-1.5-pro"
try:
    response = await client.create(messages=messages)
    print(response.content)
except Exception as e:
    print(e)

fails:

family ==  'gemini-2.5-pro':No transformer found for model family 'gemini-2.5-pro'
family ==  'gemini-1.5-pro':The capital of France is **Paris**.

[SongChiYoung]

@nicsuzor Yup! after #6213 your test case is work well~

family ==  'gemini-2.5-pro':The capital of France is **Paris**.
family == 'gemini-1.5-pro':The capital of France is **Paris**.

Thanks for the feedback!