Azure LLM evaluation - Deployment Error

[meenusel]

Describe the bug Trying to use an Azure API Key to run a LLM evaluation using UpTrain. I received a 404 error message saying that the deployment is not found. However, there is no deployment name parameter when creating the settings for the Eval LLM.

To Reproduce I followed the steps in this documentation: https://docs.uptrain.ai/llms/azure Error message: ERROR | uptrain.operators.language.llm:async_process_payload:103 - Error when sending request to LLM API: Error code: 404 - {'error': {'code': 'DeploymentNotFound', 'message': 'The API deployment for this resource does not exist. If you created the deployment within the last 5 minutes, please wait a moment and try again.'}}

[ryancole0]

I don't think this is a bug, just looks like a documentation update would be helpful. The Azure OpenAI term "deployment" usually translates to "model" in other frameworks. The following works for me:

settings = Settings(model = 'azure/{}'.format(deployment_name), 
                    azure_api_key=os.getenv('API-KEY'), 
                    azure_api_version=api_version, 
                    azure_api_base=endpoint_url)
eval_llm = EvalLLM(settings)

[terry07]

Hello there.

That library seems great, but I still face issues when I try to sync with my Azure credentials.

None of the relevant comments did help me.

Can you please check for any inconsistency?

{ "name": "ValueError", "message": "OpenAI API Key is invalid", "stack": "--------------------------------------------------------------------------- ValueError Traceback (most recent call last) Cell In[5], line 5 1 settings = Settings(model = 'azure/{}'.format('gpt-4'), 2 azure_api_key=os.getenv('AZURE_OPENAI_API_KEY'), 3 azure_api_version=os.getenv('OPENAI_API_VERSION'), 4 azure_api_base=os.getenv('AZURE_ENDPOINT')) ----> 5 eval_llm = EvalLLM(settings)

File ~/miniconda3/envs/eval/lib/python3.9/site-packages/uptrain/framework/evalllm.py:101, in EvalLLM.init(self, settings, openai_api_key) 99 response = check_openai_api_key(self.settings.openai_api_key) 100 if not response: --> 101 raise ValueError("OpenAI API Key is invalid") 103 self.executor = APIClientWithoutAuth(self.settings)

ValueError: OpenAI API Key is invalid" }

[cthompson-insight]

This is failing when you check for an OpenAI key. The check_openai_api_key only checks if the key is valid using openai.

def check_openai_api_key(api_key):
    
    import openai
    client = openai.OpenAI(api_key=api_key)
    try:
        client.models.list()
    except openai.AuthenticationError:
        return False
    else:
        return True

Needs to also use the AzureOpenAI class to do the check.

[terry07]

Great, I replaced that part with the corresponding part regarding AzureOpenAI check.

from openai import AzureOpenAI

client = AzureOpenAI(
        api_key = openai_api_key,
        api_version=openai_api_version,
        azure_endpoint=openai_azure_endpoint,
            )