[Feature request]: support for azure hosted openai models?

[mariusgarmhausen]

What do you need?

I was wondering if Azure hosted endpoints are already implemented?

[streichsbaer]

I was also looking for the Azure OpenAI option, but it's not yet supported.

@danielmiessler, I'd be happy to create a PR for that feature.

Let me know.

[GuizzyQC]

Try adding a OPENAI_BASE_URL= line in ~/.config/fabric/.env

It does complain about "Unable to fetch models" and "No OpenAI API key found" (despite having an API key too), as elsewhere in the code it does fetch a list of models from a hardcoded OpenAI URL, but outside of that it does work for me on a different OpenAI API compatible endpoint.

[bjcbusiness]

Try adding a OPENAI_BASE_URL= line in ~/.config/fabric/.env

It does complain about "Unable to fetch models" and "No OpenAI API key found" (despite having an API key too), as elsewhere in the code it does fetch a list of models from a hardcoded OpenAI URL, but outside of that it does work for me on a different OpenAI API compatible endpoint.

This doesn't work at all for me. Looking forward to proper support for Azure.

[vfarcic]

👍 for Azure OpenAI support

[AtheeSouhait]

Made it working with my Azure openai deployment, you can test: https://github.com/AtheeSouhait/fabric/tree/AtheeSouhait-AzureOpenAIonCLI

New environment variables must contain: AZURE_OPENAI_API_KEY AZURE_OPENAI_ENDPOINT AZURE_OPENAI_API_VERSION AZURE_OPENAI_MODEL for example: export AZURE_OPENAI_API_VERSION=2024-02-15-preview export AZURE_OPENAI_ENDPOINT=https://yourdeploymentendpoint.openai.azure.com/ export AZURE_OPENAI_MODEL=myGPT4-32k export AZURE_OPENAI_API_KEY=yourazureopenaideploymentapikey

[bjcbusiness]

Works for me. I set the required environment variables in the .env file. Able to list models and use a pattern.

Thank you very much.

From: AtheeSouhait @.> Sent: Saturday, June 1, 2024 5:39 PM To: danielmiessler/fabric @.> Cc: Collar, Ben (DI SW PLM LCS CF) @.>; Comment @.> Subject: Re: [danielmiessler/fabric] [Feature request]: support for azure hosted openai models? (Issue #179)

Made it working with my Azure openai deployment, you can test: https://github.com/AtheeSouhait/fabric/tree/AtheeSouhait-AzureOpenAIonCLI

New environment variables must contain: AZURE_OPENAI_API_KEY AZURE_OPENAI_ENDPOINT AZURE_OPENAI_API_VERSION AZURE_OPENAI_MODEL for example: export AZURE_OPENAI_API_VERSION=2024-02-15-preview export AZURE_OPENAI_ENDPOINT=https://yourdeploymentendpoint.openai.azure.com/ export AZURE_OPENAI_MODEL=myGPT4-32k export AZURE_OPENAI_API_KEY=yourazureopenaideploymentapikey

Reply to this email directly, view it on GitHubhttps://github.com/danielmiessler/fabric/issues/179#issuecomment-2143508728, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BHHDEZ7ECXS5RVD2DB5BUCTZFH2MDAVCNFSM6AAAAABERBCBASVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBTGUYDQNZSHA. You are receiving this because you commented.Message ID: @.@.>>

[stefanstranger]

Hi @AtheeSouhait, I followed your instructions and copied your code changes but I don't see the model being showed when running fabric --listmodels.

What else do I need to change to support Azure OpenAI?

Thanks for your help. /Stefan

[AtheeSouhait]

Hi @stefanstranger , indeed the update don't modify output of --listmodel so it is not included in the list. But your setting automaticaly uses your AZURE open ai model. By the way, if you want to use OPENAI model and not the Azure; comment the AZURE_OPENAI_API_KEY line of your .env so it is not taken as default. I should have also adapted this "listmodel" part of the code. I'll do if I have the time but as the project will move to "go" language implementation, I'm not sure it worth the effort, what do you think?

[stefanstranger]

Hi @AtheeSouhait,

Thanks for your quick response 🙏.

I've commented out the OPENAI_API_KEY for now to be sure it will use the AZURE_OPENAI_API_KEY.

When I now try use Fabric I get the following error.

echo "What is the Capital of the Netherlands?" | fabric --stream --pattern ai
Error: 'NoneType' object has no attribute 'chat'
'NoneType' object has no attribute 'chat'

When running the following curl I just get a response:

curl "https://<instancename>.openai.azure.com/openai/deployments/chatdemo/chat/completions?api-version=2024-02-15-preview" \
  -H "Content-Type: application/json" \
  -H "api-key: <enter api key>" \
  -d "{
  \"messages\": [{\"role\":\"system\",\"content\":\"What is the capital of the Netherlands?\"}],
  \"max_tokens\": 800,
  \"temperature\": 0.7,
  \"frequency_penalty\": 0,
  \"presence_penalty\": 0,
  \"top_p\": 0.95,
  \"stop\": null
}"

Something still is not configured correctly.

What am I missing? Is there a way to list this model?

Added note:

• Running this on WSL Ubuntu.

[AtheeSouhait]

@stefanstranger did you get my repo's code? because I can"t see my changes any more in the official version. Or you can see what I changed in the utils.py file and report in the repo you cloned.

[stefanstranger]

Yes I copied the raw content from 'your' utils.py file to my local repo.

[AtheeSouhait]

Not working when I use --stream but working without. Is it working when you use openai and not azure Open ai (comment the AZURE_OPENAI_API_KEY of env) ?

[stefanstranger]

Tried without using --stream switch but that is also giving me the same error.

Error: 'NoneType' object has no attribute 'chat' 'NoneType' object has no attribute 'chat'

It is working when I use OpenAI but not when I use Azure OpenAI.

Very strange that I cannot get this working with Azure OpenAI.

[MrPewPewLaser]

@stefanstranger I got it to work for me. The problem seems to be that his Code is far behind the original repo that requires newer package versions. So I cloned his entire repo and forced a new install of fabric using pipx install . --force

Hope it helps. :)

[stefanstranger]

Hi @MrPewPewLaser thanks for your help.

Will try your suggestions and let you know if that works.

Regards, Stefan

[AtheeSouhait]

Thanks @MrPewPewLaser, @stefanstranger : I also re-tested : resinstallation of fabric from https://github.com/danielmiessler/fabric (cloned, put my utils.py, and continue all installation steps + add Azure environment variables). And after the installation it worked (with gpt4 model in Azure , I didn't try 3.5 or other models yet). I know it is not perfect as --stream seams not to be supported. I hope it will work for you.

[stefanstranger]

I got it to work. One more lesson learned during the setup of the Azure OpenAI configuration is that is need the Azure OpenAI deployment name and NOT the Model name in the configuration.

[AtheeSouhait]

@stefanstranger @MrPewPewLaser Great, I'm working on a dockerfile to ease the installation, setup, and the usage. I let you know.