feat(frontend): Improve models input UI/UX in settings

[amanape]

What is the problem that this fixes or functionality that this introduces? Does it fix any open issues? The models input dumps all the raw LiteLLM IDs that cause slight performance issues and overwhelm the user with options.

---

Give a summary of what the PR does, explaining any non-trivial design decisions

Default behaviour

This PR breaks down the input into a Provider input and Models input. Given the provider, a filtered models list will be available for the user to choose.

• Changed default LLM_MODEL to openai/gpt-4o from gpt-4o

The user can see the "Verified" providers and models that we recommend and know work well with OpenHands.

utils/verified-models.ts

export const VERIFIED_PROVIDERS = ["openai", "azure", "anthropic"];
export const VERIFIED_MODELS = ["gpt-4o", "claude-3-5-sonnet-20240620-v1:0"];

If preferred, the user can set a custom LLM model.

• New settings were added, CUSTOM_LLM_MODEL and USING_CUSTOM_LLM. That way, the socket class sends the appropriate LLM with the LLM_MODEL key if settings.USING_CUSTOM_LLM is true.

Tests

• ~85 - 90%+ test coverage

Notes

• Need to confirm which are the verified providers and models we want to set and display to the user.
• Need to confirm that the new way of setting custom LLM models is OK
• Since LiteLLM does not maintain consistency between supported models (some are not returned at all and just say they support it by append some-provider/), most are dumped in an other provider. Need to confirm if it is OK, which ones do we want to "pull out", and if there are alternative ways to handle this (TBH considering going to the LiteLLM repo and making the changes for consistency myself)

TODO

• [x] Add a switch to allow the user to switch between LiteLLM models and a custom model
• [ ] ~~Investigate providers that offer custom models~~
• [x] Input validation
• [x] Set verified models and providers
• [x] Bug fixes

Accidentally removed #3514 after re-forking

[amanape]

@neubig Could confirm the verified providers and models we want to display?

export const VERIFIED_PROVIDERS = ["openai", "azure", "anthropic"];
export const VERIFIED_MODELS = ["gpt-4o", "claude-3-5-sonnet-20240620-v1:0"];

[tobitege]

Is OpenRouter still selectable as provider? For reference: https://openrouter.ai/models

[amanape]

@tobitege Yes, looks like we do:

The thing I can't be certain is if all the models OpenRouter offers through LiteLLM are under this provider. It could be the case that there are additional models LiteLLM does NOT list/return, or they do but without the provider (e.g., some-model instead of openrouter/some-model). In those cases, I will need to do a little hardcoding stuff as I did for OpenAI

[tobitege]

@tobitege Yes, looks like we do:

Thanks! Yes, OpenRouter could add new models any day, but usually litellm is usually pretty fast in adding support for new ones. I'd not restrict any selection a user may do with OpenRouter, it'll either work or not, to be blunt. 😬

[amanape]

I'd not restrict any selection a user may do with OpenRouter, it'll either work or not, to be blunt.

Yes I'm trying not to restrict anything here. Actually the thing that this PR does remove is the ability to input a custom model given a provider (e.g., openrouter/some-random-model-maybe-not-in-docs). Do you think this is OK? I can't see the reasons why the user would want to do such a thing in the first place

[tobitege]

I'd not restrict any selection a user may do with OpenRouter, it'll either work or not, to be blunt.

Yes I'm trying not to restrict anything here. Actually the thing that this PR does remove is the ability to input a custom model given a provider (e.g., openrouter/some-random-model-maybe-not-in-docs). Do you think this is OK? I can't see the reasons why the user would want to do such a thing in the first place

I think it'd be ok. The typing in the box always felt wonky anyways.

[tobitege]

So users could still enter anything they want via the Custom option, right? Like for Ollama etc.?

[enyst]

I'd not restrict any selection a user may do with OpenRouter, it'll either work or not, to be blunt.

Yes I'm trying not to restrict anything here. Actually the thing that this PR does remove is the ability to input a custom model given a provider (e.g., openrouter/some-random-model-maybe-not-in-docs). Do you think this is OK? I can't see the reasons why the user would want to do such a thing in the first place

@amanape There is this tricky thing where liteLLM interprets the model names like "openai/something" as a hint to route the completion call to an openai-compatible endpoint, which they build according to base_url or provider-specific url, and ending with /v1, in other words they build some API URL which they know accepts messages in OpenAI format, because the user added "openai" prefix. (In that case it also makes sure that the parameters are in the right format, I think)

I don't know if that has relevance for openrouter. I know our users needed it in some cases, like CommandR and/or Cohere where the prefix worked, its missing did not.

My point is that typing in the box was necessary to add /openai to some prefixes. Maybe it's fine if it's still possible via custom model, though.

Sorry for typing in a closed issue, just a quick thought and I'll take it these days elsewhere for further thinking/testing.