continue Autocomplete using Azure Open AI throws error $.stop' is invalid

Before submitting your bug report

[X] I believe this is a bug. I'll try to join the Continue Discord for questions
[X] I'm not able to find an open issue that reports the same bug
[X] I've seen the troubleshooting guide on the Continue Docs

Relevant environment info

- OS: Windows 10 x64 10.0.19045
- Continue: 0.9.7919
- IDE: VSCode 1.88.0-insider (d73fa8b14a6c873958d00a7d7ad13fcb540a052c)
- Model: Azure OpenAI Chat GPT 3.5 Turbo 0301

Description

While using tab autocomplete, Continue throws error: Error generating autocomplete response: Error: HTTP 400 model_error from https://XXX.openai.azure.com/openai/deployments/gpt-35-turbo/completions?api-version=2023-05-15 { "error": { "message": "'$.stop' is invalid. Please check the API reference: https://platform.openai.com/docs/api-reference.", "type": "invalid_request_error", "param": null, "code": null } } This error doesn't occur chat, just autocomplete

To reproduce

Obtain Azure OpenAI Chat GPT 3.5 Turbo 0301 url and secret. Make sure your model support autocomplete.
Update Continue config chat to use custom Azure OpenAI Chat GPT model with given configuration
Test chat, to make sure it works fine
Update Continue config - section tabAutocompleteModel with same model configuration as used and verified above.
Try to write code.
Continue throws error for each autocomplete request.
API version changes or ChatGPT model change doesn't fix error.

used model configuration: { "title": "Azure OpenAI 3.5 Turbo", "provider": "openai", "model": "gpt-3.5-turbo", "apiBase": "https://XXX.openai.azure.com/", "engine": "gpt-35-turbo", "apiVersion": "2023-07-01-preview", "apiType": "azure", "apiKey": "XXX" },

Log output

Error generating autocomplete response: Error: HTTP 400 model_error from https://XXX.openai.azure.com/openai/deployments/gpt-35-turbo/completions?api-version=2023-05-15 { "error": { "message": "'$.stop' is invalid. Please check the API reference: https://platform.openai.com/docs/api-reference.", "type": "invalid_request_error", "param": null, "code": null } }

Mar 05 '24 15:03 mabocek

same here with GPT-4-1106 Preview on Azure Endpoint

Mar 08 '24 07:03 StefanRaab

Yes same error for me with GPT4 model and Azure Endpoint

Mar 11 '24 15:03 FPrz

It seems to me like an API version mismatch. Have you tried using "apiVersion": "2023-05-15" in your config?

Mar 13 '24 13:03 jakubsemerak

It seems to me like an API version mismatch. Have you tried using "apiVersion": "2023-05-15" in your config?

Yep, I tried multiple API versions. Changing API version doesn't fix the issue for me. Did it work for you?

Mar 14 '24 15:03 mabocek

No it did not work, I have still the same error with different "apiVersion", including the 2023-05-15 Thanks in advance

Mar 15 '24 08:03 FPrz

@FPrz @mabocek @jakubsemerak @StefanRaab This must be because Azure is throwing an error when more than 4 stop words are sent. I've found that only certain servers actually enforce this. Just added a check to truncate the list of stop words when Azure is being used.

But also I feel it's important to note that you very likely will not want to use GPT for autocomplete. All models that are made for autocomplete are trained with an extremely specific prompt format and almost their entire training is dedicated to autocomplete. GPT on the other hand is not trained for autocomplete and has no such format. You can see here how we've attempted to prompt it to output text given a prefix, but this is much less reliable. GPT is also slower

I would recommend using starcoder or deepseek-coder. In practice it just turns out that a large model doesn't help much with autocomplete (most of the state-of-the-art models are no more than 7b parameters).