Cannot setup an Azure OpenAI model

[kedar-1]

Before submitting your bug report

• [ ] I believe this is a bug. I'll try to join the Continue Discord for questions
• [ ] I'm not able to find an open issue that reports the same bug
• [ ] I've seen the troubleshooting guide on the Continue Docs

Relevant environment info

- OS:Windows
- Continue:last
- IDE: VSCode
- Model: gpt-4o
- config.json:

Description

How to setup an Azure Open AI model?

What is Engine? Where I can grab this value in Azure portal/AI Studio?

What is API Base? Where I can grab this value in Azure portal/AI studio?

Please create a documentation not only what must be inserted into config file but WHERE I can find the required value.

Thanks

To reproduce

1. Open a Continue.dev.
2. Click on + to Add model.
3. Select an Azure

Log output

No response

[geekloper]

I agree that the documentation for the Azure OpenAI service could be clearer. Here's how I configured mine:

Configuration in config.json

{
  "models": [{
    "title": "Azure OpenAI",
    "provider": "azure",
    "model": "<model>",
    "engine": "<deployment_id>",
    "apiBase": "<azure_open_ai_endpoint>",
    "apiType": "azure",
    "apiKey": "<azure_open_ai_key>"
  }...

Explanation of Configuration Parameters

• <model>: You can find this in the deployment section of Azure OpenAI Studio (e.g., gpt35turbo).
• <deployment_id>: This is the name you chose for the model in Azure OpenAI Studio, also located in the deployment section.
• <azure_open_ai_endpoint> & <azure_open_ai_key>: This can be found under the Azure OpenAI resource in the Azure portal:
  • Navigate to Resource Management
  • Select Keys and Endpoint

[ChristophRaab]

I agree with @geekloper. Thats also how i configured it. However, i looked up the current ModelDescription-Schema but i think the "engine", "apiVersion" and "apiType" may missing. If i follow the model description, i am not able to come up with a working azure models.

See: https://github.com/continuedev/continue/blob/0ef443cafc00eb44223bb34c81e52c3b1910eb3e/core/index.d.ts#L714-L726

[kedar-1]

Thanks @geekloper. With your help I was able to make a Continue.dev extension to work with GPT-4o model. I'll try to create a PR as the documentation on project site its not user friendly.

BTW: Setting up TabAutocomplettion to llama3.1 running on ollama local I have a nice, smart and cheap Copilot :-)

[alessap]

Is it possible to use a self-hosted LLM? Let's say I have a GPT-4o hosted in my own cloud service (Azure)?

[ctwardy]

Is it possible to use a self-hosted LLM? Let's say I have a GPT-4o hosted in my own cloud service (Azure)?

Yes. Until today's stable release, we ran it with our API base in apiBase, and our deployment in engine. However, as of today it is mistakenly using model (gpt-4o) instead of engine (we used "gpt4o", no dash) when constructing the API endpoint.

Update:

That was fixed within a couple of weeks so that the config correctly used deployment to construct the endpoint.

[Jamie-BitFlight]

Additional quirk:

If you configure it with anything in the "model" field, like gpt-3-turbo, gpt4, gpt4o, gpt-4, gpt-4o, o1-mini - even if your deployment only has a gpt-3 model, it processes the request fine. If you configure it with nothing in the "model" or leave out the model field. Then it just hangs when attempting to get a response.

So I think the 'model' field is for internal use to define the capabilities rather than a requirement for the web query.

[danielbelemduarte]

I'm trying to setup AzureOpenAI o3-mini with that same payload, works just fine with gpt4 but not with o3-mini. Any comment on how to fix this ?

[d4nielmeyer]

To make AzureOpenAI o3-mini work I guess someone needs to update azure.ts as done here for openai.ts as 03-mini e.g. does not accept max_tokens any more but expects max_completion_tokens instead: https://github.com/continuedev/continue/pull/3914/files

[Patrick-Erichsen]

@d4nielmeyer thanks for linking that PR, I believe that is what's needed to get o3 working for Azure. I'm not sure how soon we'll be able to prioritize this so if anyone has interest in opening up a PR that would be much appreciated!