[BUG] Allow user to manually define context wnidow for unknown models

[lenaxia]

Description

Conversation from discord:

Lenaxia — 05/09/2024 11:01 PM Is there a way to forcefully define context windows for unknown models?

I've got BigAGI pointed to AWS Bedrock by using LiteLLM as a proxy by creating a OpenAI #2 service in BigAGI. However I defined some custom models like for Claude Sonnet and Mixtral 8x7b, but these are showing in the BigAGI models window as ?: Unknown, please let us know the ID. Assuming a 4097 context window size. Obviously Claude has much more than a 4097 context window, I just can't figure out a place to change it for unknown models.

Enrico — 05/10/2024 4:09 AM We don't have that feature. Would be a nice addition indeed, and something not that hard to do. As the API does not communicate the name/description/window, there are 2 possibilities: either the developer hardcodes the model_id -> metadata mapping, or we give the user a UI to change the context window (and max output window) numbers. You should create a bug about it - we have a bug about ignoring context windows and never setting them in the UI or model calls, but not a ticket to manually specify the context windows.

Bascially, in the model management window, allow the context length to be editable by the user so they could change it manually if they need to.

Device and browser

N/A

Screenshots and more

N/A

Willingness to Contribute

• [ ] 🙋‍♂️ Yes, I would like to contribute a fix.

[enricoros]

See #531