RA.Aid icon indicating copy to clipboard operation
RA.Aid copied to clipboard

ReAct Agent backend is no longer used for OpenRouter Provider + Possible Model Misconfiguration

Open ariel-frischer opened this issue 1 year ago • 1 comments

If we use --model "anthropic/claude-3.7-sonnet" --provider "openrouter" then we don't default to create_react_agent anymore due to new changes in agent_utils.py file. Specifically logic around: uses_react_backend seems off.

In two places: create_agent and build_agent_kwargs I believe we may be fetching the 'model_name' and provider incorrectly.

  • https://github.com/ai-christianson/RA.Aid/blob/master/ra_aid/agent_utils.py#L77
  • https://github.com/ai-christianson/RA.Aid/blob/master/ra_aid/agent_utils.py#L140

For example

    provider = get_config_repository().get("provider", "anthropic")
    model_name = get_config_repository().get("model", "")

    # Get the model parameters from models_params
    from ra_aid.models_params import models_params, AgentBackendType

    provider_config = models_params.get(provider, {})
    model_config = provider_config.get(model_name, {})

    # Check if this model uses the REACT backend (which is what we're configuring kwargs for)
    uses_react_backend = (
        model_config.get("default_backend", DEFAULT_AGENT_BACKEND)
        == AgentBackendType.CREATE_REACT_AGENT
    )

Does not take into account the variations of --research-model, --research-provider, --planner-model, --planner-provider. So it may be pulling the wrong model/provider in two places.

Additionally, AgentBackendType.CREATE_REACT_AGENT is not specified in "default_backend" of models_params dict for openrouter claude. So openrouter versions of claude do not have that backend set, causing us to no longer use create_react_agent for those models when it did previously. I propose we should just do a simple check to see if "model_name" has "claude" string in it. Use create_react_agent if so, but if default_backend IS specified in model_params we can override it with that value.

ariel-frischer avatar Mar 17 '25 23:03 ariel-frischer

If we use --model "anthropic/claude-3.7-sonnet" --provider "openrouter" then we don't default to create_react_agent anymore due to new changes in agent_utils.py file. Specifically logic around: uses_react_backend seems off.

We now have this in models_params.py:

DEFAULT_AGENT_BACKEND = AgentBackendType.CIAYN

The model config should look like this:

        "claude-3-7-sonnet-20250219": {
            "token_limit": 200000,
            "supports_temperature": True,
            "supports_thinking": True,
            "max_tokens": 64000,
            "default_temperature": 1.0,
            "latency_coefficient": DEFAULT_BASE_LATENCY,
            "default_backend": AgentBackendType.CREATE_REACT_AGENT
        },

We do not have params in there for using sonnet via openrouter/openai-compatible, so adding that would be the proper fix.

The idea is to get rid of all the special cases code --we shouldn't have any more "if_is_sonnet" type logic --it should all be done via models params.

I propose we should just do a simple check to see if "model_name" has "claude" string in it. Use create_react_agent if so, but if default_backend IS specified in model_params we can override it with that value.

Trying to get away from having a bunch of special cases in the code.

Does not take into account the variations of --research-model, --research-provider, --planner-model, --planner-provider. So it may be pulling the wrong model/provider in two places.

Yeah this should be fixed/improved 👍

ai-christianson avatar Mar 19 '25 18:03 ai-christianson