Allow user to set AI assist provider

[hongbo-miao]

Description

We have claude-3-7-sonnet model behind LiteLLM https://github.com/BerriAI/litellm which exposes OpenAI format API.

However, currently based on

If the model starts with "claude-", we will use your Anthropic API key. If the model starts with "gemini-", we will use your Google Al API key. Otherwise, we will use your OpenAI API key.

it will use Anthropic API instead of OpenAI API.

Suggested solution

It would be great to allow user to set AI assist provider, thanks! ☺️

Alternative

No response

Additional context

No response

[mscolnick]

@hongbo-miao if you are blocked, i don't mind putting a regex if the base_url contains litellm. Our logic is based on guessing via model. Doesn't seem too crazy to also look at the base_url.

Long term we can add a dropdown to choose a provider, but would require a bit more work. Here is the current code.

def get_completion_provider(
    config: AnyProviderConfig, model: str
) -> CompletionProvider[Any, Any]:
    if model.startswith("claude"):
        return AnthropicProvider(model, config)
    elif model.startswith("google") or model.startswith("gemini"):
        return GoogleProvider(model, config)
    else:
        return OpenAIProvider(model, config)

Would you be open to making the change?

[hongbo-miao]

Hi @mscolnick I think guessing based on base_url or model are kind of tricky. And the logic may become more and more complex.

Here is how Cline set up which may be a good reference:

If some UI allows us to select provider like Cline's approach, that would be awesome, thanks!

[mscolnick]

Sounds good, this looks like a much better approach.

[sanzoghenzo]

Hi, I started to implement the changes for #4460, and as suggested I'm introducing an ai.provider_type configuration option (focusing on the back-end for now).

Should we totally get rid of the model based selection and the require the provider_type, or should we keep the current logic as a fallback?

should we also move the model option up to the ai section (AiConfig class)? Obviously I would still check for ai.open_ai.model to avoid breaking changes, maybe displaying a deprecation warning for a while.

[mscolnick]

@sanzoghenzo - maybe we can keep the logic for now as a fallback.

the default provider_type could be an empty string "", and if its empty, then we use the old logic. the frontend can allow you to set a specific provider, but won't let you set back to an empty string.

---

for moving ai.open_ai.model to ai.model. I think that makes sense...just thinking out loud, i am now wondering if each config should have their own model in case you want to switch providers easily and keep one model per provider. (i personally don't do that)

[sanzoghenzo]

for moving ai.open_ai.model to ai.model. I think that makes sense...just thinking out loud, i am now wondering if each config should have their own model in case you want to switch providers easily and keep one model per provider. (i personally don't do that)

I also hit a roadblock regarding the base_url for Azure: should we reuse the current base_url (currently only for openai) or add a new input?

This is becoming more and more difficult for my (front end) coding level, and I suppose it needs a through review by the steering committee!

[mscolnick]

@sanzoghenzo happy to help out on the frontend once you put up your PR.

for azure, I think we can create a new entry for ai.azure.base_url, and fallback to the open_ai.base_url to start.

[vanillechai]

@hongbo-miao

As a workaround: Rename the models in LiteLLM. Make sure that they don't contain any substrings "gemini" or "claude", and marimo will work just fine with LiteLLM.

[mscolnick]

This is now fixed by https://github.com/marimo-team/marimo/pull/6018 (which will be in the next release 0.15.0). Providers are encoded in the model ID: azure/gpt-4o, openai/gpt-4o