not able to connect llamaguard with nemoguardrail

[shashankc28]

hi i am not able to connect my llamaguard api with nemoguardrail . below are the configs i have used . please help !!!

config :

models:
  - type: main
    engine: openai
    model: gpt-3.5-turbo-instruct

  - type: llama_guard
    engine: vllm_openai
    parameters:
      openai_api_base: "https://localhost:8001/v1"
      model_name: "meta-llama/LlamaGuard-7b"

rails:
  input:
    flows:
      - llama guard check input

  output:
    flows:
      - llama guard check output

chat.py file :

import os
from langchain_community.chat_models import AzureChatOpenAI
from nemoguardrails import LLMRails, RailsConfig

import pandas as pd

# Define LLM and parameters to pass to the guardrails configuration
chat_model = AzureChatOpenAI(****** )

# Load configuration
config = RailsConfig.from_path("./config")

# Configuration of LLMs is passed
app = LLMRails(config=config, llm=chat_model)

 # sample user input
 new_message = app.generate(messages=[{
     "role": "user",
     "content": "what is the length of the atm pin?"
}])

prompt file same as shown in the example.

command used to deploy model : python -m vllm.entrypoints.openai.api_server --model meta-llama/LlamaGuard-7b --port 8005 --dtype float16 --max-model-len 1500

logs show this :

/home/rndadmin/nemoguardrail/guard/lib/python3.10/site-packages/langchain_core/_api/deprecation.py:119: Lang          ChainDeprecationWarning: The class `AzureChatOpenAI` was deprecated in LangChain 0.0.10 and will be removed           in 0.3.0. An updated version of the class exists in the langchain-openai package and should be used instead.           To use it run `pip install -U langchain-openai` and import as `from langchain_openai import AzureChatOpenAI          `.
  warn_deprecated(
/home/rndadmin/nemoguardrail/guard/lib/python3.10/site-packages/langchain_community/chat_models/azure_openai          .py:167: UserWarning: As of openai>=1.0.0, Azure endpoints should be specified via the `azure_endpoint` para          m not `openai_api_base` (or alias `base_url`). Updating `openai_api_base` from https://autogenaillmapi.opena          i.azure.com/ to https://autogenaillmapi.openai.azure.com/openai.
  warnings.warn(
/home/rndadmin/nemoguardrail/guard/lib/python3.10/site-packages/langchain_community/chat_models/azure_openai          .py:174: UserWarning: As of openai>=1.0.0, if `deployment_name` (or alias `azure_deployment`) is specified t          hen `openai_api_base` (or alias `base_url`) should not be. Instead use `deployment_name` (or alias `azure_de          ployment`) and `azure_endpoint`.
  warnings.warn(
Entered verbose mode.
Fetching 5 files: 100%|████████████████████████████████████████████████████| 5/5 [00:00<00:00, 69905.07it/s]
Event UtteranceUserActionFinished {'final_transcript': 'what is the length of the atm pin?'}
Event StartInternalSystemAction {'uid': '057f...', 'action_name': 'create_event', 'action_params': {'event':
{'_type': 'StartInputRails'}}, 'action_result_key': None, 'action_uid': 'f514...', 'is_system_action': True}
Executing action create_event
Event StartInputRails {'uid': '2728...'}
Event StartInternalSystemAction {'uid': 'ca3c...', 'action_name': 'create_event', 'action_params': {'event':
{'_type': 'StartInputRail', 'flow_id': '$triggered_input_rail'}}, 'action_result_key': None, 'action_uid':
'9536...', 'is_system_action': True}
Executing action create_event
Event StartInputRail {'uid': '995b...', 'flow_id': 'llama guard check input'}
Event StartInternalSystemAction {'uid': '23a0...', 'action_name': 'llama_guard_check_input',
'action_params': {}, 'action_result_key': 'llama_guard_response', 'action_uid': '907d...',
'is_system_action': False}
Executing action llama_guard_check_input
Event hide_prev_turn {}
Total processing took 0.01 seconds. LLM Stats: 0 total calls, 0 total time, 0 total tokens, 0 total prompt
tokens, 0 total completion tokens, [] as latencies
{'role': 'assistant', 'content': "I'm sorry, an internal error has occurred."}

[drazvan]

@prasoonvarshney : can you help debug this? Thanks!

[nikitha-krishnan]

Same issue, why there is not response

18:00:19.109 | Event UtteranceUserActionFinished | {'final_transcript': 'Hello! What can you do for me?'} 18:00:19.112 | Event StartInternalSystemAction | {'uid': '8774...', 'action_name': 'create_event', 'action_params': {'event': {'_type': 'StartInputRails'}}, 'action_result_key': None, 'action_uid': '7612...', 'is_system_action': True} 18:00:19.114 | Executing action create_event 18:00:19.115 | Event StartInputRails | {'uid': '9d3d...'} 18:00:19.117 | Event StartInternalSystemAction | {'uid': '5a17...', 'action_name': 'create_event', 'action_params': {'event': {'_type': 'StartInputRail', 'flow_id': '$triggered_input_rail'}}, 'action_result_key': None, 'action_uid': '1086...', 'is_system_action': True} 18:00:19.119 | Executing action create_event 18:00:19.121 | Event StartInputRail | {'uid': '3459...', 'flow_id': 'content safety check input $model=llama-guard3:1b'} 18:00:19.123 | Event StartInternalSystemAction | {'uid': '3103...', 'action_name': 'content_safety_check_input', 'action_params': {}, 'action_result_key': 'response', 'action_uid': 'ad17...', 'is_system_action': False} 18:00:19.125 | Executing action content_safety_check_input 18:00:19.128 | Event hide_prev_turn | {} 18:00:19.130 | Total processing took 0.02 seconds. LLM Stats: 0 total calls, 0 total time, 0 total tokens, 0 total prompt tokens, 0 total completion tokens, [] as latencies