AzureOpenAI is not supported

[visheshgitrepo]

Hi @drazvan

I dont see Azurechatopenai supports parameters like top_p, presence_penalty,frequency_penalty. But its supported by AzureOpenAI, but this is supported in Nemo LLM providers. Please clarify, if we can use, AzureOpenAI instead of AzureChatOpenAI. Im getting below error when using AzureOpenAI.

error: Error while execution self_check_input: Error code: 404 - {'statusCode': 404, 'message': 'Resource not found'}

And if we provide parameters like temperature, top_p, etc in configuration files, will it upate the LLM for below code..

config = RailsConfig.from_path("./config") rails = LLMRails(config,llm=chat_model)

[visheshgitrepo]

Just small correction, Nemo doesnt support AzureOpenAI

[drazvan]

@visheshgitrepo : can you provide more detail on the configuration (i.e., the content of config.yml)? Any custom parameters can be specified inside the parameters key when configuring the main LLM (https://github.com/NVIDIA/NeMo-Guardrails/blob/develop/docs/user_guides/configuration-guide.md#the-llm-model)

models:
- type: main
  engine: xyz
  model: abc
  parameters:
    param_1: value_1
    ...

[visheshgitrepo]

@drazvan Here is how Im passing the model right now. A workaround. But What I want is, these paramters of Azurechatopenai must be updated with the values from config.yml. Note: Deployment_name and Model_name of Azure might vary, like we have values like gpt-35 instead of 3.5.

Code

Define LLM and parameters to pass to the guardrails configuration

chat_model = AzureChatOpenAI( temperature = 1, max_tokens = 1024, openai_api_type="azure", openai_api_version=api_version, api_key=azure_openai_key, deployment_name=azure_openai_model, azure_endpoint=azure_openai_endpoint, model_kwargs = { "presence_penalty": 0, "frequency_penalty": 0, "top_p": top_p} ) llmrails = LLMRails(RailsConfig.from_path(str(rails_config_path)), llm=chat_model)

Config.yml

models:

• type: main engine: openai model: gpt-35-turbo-16k

I tried adding parameters like below but didnt work, unable to make connection

models:

• type: main engine: azure model: gpt-3.5 parameters: temperature: 1 max_tokens: 1024 api_version: version azure_endpoint: url deployment_name: gpt-35-turbo openai_api_type: azure model_kwargs: presence_penalty: 0 frequency_penalty: 0

And created rail without chat_model LLMRails(RailsConfig.from_path(str(rails_config_path))

[visheshgitrepo]

@drazvan One more thing is, my primary focus is to use gpt-3.5 for input, output rail. And gpt-4 for generate_bot_message only, remaining user_intent and bot_intent should also from gpt-3.5. I tried multiple ways not working How we can do that?

[drazvan]

@visheshgitrepo: if you set the type of the model to generate_bot_message_llm and set it to use GPT-4, and leave the 3.5 as the main, it should work. Let me know if it doesn't.

models:
- type: generate_bot_message_llm
  engine: xyz
  model: abc
  parameters:
    param_1: value_1
    ...

[visheshgitrepo]

@drazvan Its not working, the problem here is. As Im using AzureChatOpenAI due to azureopenaikey. without passing chat_model externally to below line, its not working at all. chat_model = AzureChatOpenAI(.....) llmrails = LLMRails(RailsConfig.from_path(str(rails_config_path)), llm=chat_model)

So, I need to define deployment_name i.e, gpt-35-turbo in chat_model, even If I set gpt-4 in config for bot message,as you suggested above, still its taking the deployment model used in chat_model.

If I try to remove this llm passing externally to LLMRails like below. I get connection error at self_check_input llmrails = LLMRails(RailsConfig.from_path(str(rails_config_path)))

Im using engine as azure, and model is gpt-35-turbo(this is deployment name), in config im passing model as gpt-3.5, as in code of Nemo its validating with gpt-3.5/gpt-4 only..

How to resolve this?

[SivaNithishKumar]

@visheshgitrepo Still facing the error ???

I got the same error changed my model name and added the deployment name inside the parameters of the config file and its working for me