Completion model provider should be traitlet-configurable

[krassowski]

trafficstars

@krassowski Will it also be better if we can set something like default_completion_model and default_chat_model when running jupyter lab --config using config.json? Previously, we can only do something like:

{
    "AiExtension": {
        "default_language_model": "Self-Hosted:deepseek-coder-6.7b-instruct",
        "allowed_providers": "Self-Hosted",
        "model_parameters": {
            "Self-Hosted:deepseek-coder-6.7b-base": {},
            "Self-Hosted:deepseek-coder-6.7b-instruct": {}
        },
        "default_api_keys":{}
    }
}

It may be better to modify the extension.py also. Thank you.

Originally posted by @yuhon0528 in https://github.com/jupyterlab/jupyter-ai/issues/711#issuecomment-2104050127

[yuhon0528]

Also, since we separated into two models (chat and completion), the API keys will most likely be different. However, for example if I use OpenAI Compliable APIs for both chat and completion models, there will only be one OPEN_API_KEY field for authentication.

How can I authenticate both models with different API keys with only one field? Thanks.

[krassowski]

For now, we would probably only want to only add default_completion_model (rather than rename default_language_model to default_chat_model) to avoid breaking compatibility.

[krassowski]

The API keys are a key-value mapping so maybe that is not a problem? If you use cohere for chat you would set COHERE_API_KEY, if you use openai for completions you would set OPENAI_API_KEY, so the default_api_keys would be:

"default_api_keys": {
  "COHERE_API_KEY": "MY_KEY_1",
  "OPENAI_API_KEY": "MY_KEY_1"
}

and respective providers would take the key they need. Does it makes sense?

[yuhon0528]

But in fact is that I am using two models that are both with OpenAI compatible APIs, that is:

{
    "AiExtension": {
        "default_language_model": "Self-Hosted:deepseek-coder-6.7b-instruct",
        "allowed_providers": "Self-Hosted",
        "model_parameters": {
            "Self-Hosted:deepseek-coder-6.7b-base": {},
            "Self-Hosted:deepseek-coder-6.7b-instruct": {}
        },
        "default_api_keys":{
            "OPENAI_API_KEY": "deepseek-coder-6.7b-base-key",
            "OPENAI_API_KEY": "deepseek-coder-6.7b-instruct-key"
        }
    }
}

When we use those self-hosting models for chat and completions, most likely will use something like vllm to provide OpenAI compatible API Servers.

Hence, that is the fact that both completion and chat model will be OpenAI API like.

[dcieslak19973]

Hoping to help get some momentum on this issue. A link to my comments on a related/duplicate issue:

https://github.com/jupyterlab/jupyter-ai/issues/953#issuecomment-2332151326

TL;DR - it does not currently seem possible to configure the Completions solely from the jupyter_jupyter_ai_config.json file

[krassowski]

This has so many upvotes, adding to v3.0 for consideration.

[dlqqq]

WOW, this issue has a lot of upvotes! Very sorry about the very late check-in, I must have missed this issue when we got busy last year. I just found this issue again while sorting through our issue board today.

Good news however; @srdas has opened a PR to add a trait for the inline completion model! See #1303. We can backport this to 2.x so users can enjoy this feature as soon as possible, without waiting for Jupyter AI v3.0.0. 🤗

I'll work with Sanjiv more tomorrow to see when we can get this merged & released. 👍