NeMo-Guardrails bug: Bot fails to execute 'denied_topic_check' action despite defined YAML and Colang flows

bug: Bot fails to execute 'denied_topic_check' action despite defined YAML and Colang flows

Open mohilmakwana31 opened this issue 8 months ago • 3 comments

Did you check docs and existing issues?

[x] I have read all the NeMo-Guardrails docs
[x] I have updated the package to the latest version before submitting this issue
[x] (optional) I have used the develop branch
[x] I have searched the existing issues of NeMo-Guardrails

Python version (python --version)

Python 3.11

Operating system/version

Docker container on AWS

NeMo-Guardrails version (if you must use a specific version and not the latest

No response

Describe the bug

The bot fails to execute the denied_topic_check action despite being defined in both the YAML configuration and Colang flows. Instead of properly blocking denied topics, the bot returns an error:

({'role': 'assistant', 'content': "Action 'denied_topic_check' not found."}, {'last_user_message': None, 'last_bot_message': "Action 'denied_topic_check' not found.", ...})

Steps to Reproduce

Deploy the bot with the following YAML configuration:
- Defined rails.config includes denied_topic_check under input.flows.
- Defined prompt for denied_topic_check.
Deploy Colang logic with denied topic check flow:
- Executes denied_topic_check.
- If $allowed is False, bot refuses to generate a response.
Ask a question that should trigger the denied_topic_check, e.g.,
```
Which type of bullet is used in AK-47?  
```
Observe the response:
- Expected: The bot refuses to answer.
- Actual: The bot returns "Action 'denied_topic_check' not found."

Expected Behavior

The bot should correctly execute the denied_topic_check flow and refuse to answer when a denied topic is detected.

Actual Behavior

The bot fails with an error stating that the action denied_topic_check is not found, even though it is defined in the configuration.

Relevant Configuration Details

YAML (`rails` section)

models:
  - type: main
    engine: openai
    model: gpt-4o-mini

instructions:
  - type: general
    content: |
      You are a helpful assistant that can answer given questions. Bot should not go off-topic.

rails:
  config: 
    sensitive_data_detection:
      input:
        entities:
          - IN_PAN
          - IN_PASSPORT
          - PHONE_NUMBER
          - EMAIL_ADDRESS
          - DATE_TIME
      output:
        entities:
          - IN_PAN
          - IN_PASSPORT
          - PHONE_NUMBER
          - EMAIL_ADDRESS
          - DATE_TIME

  input:
    flows:
      - jailbreak detection heuristics
      - self check input
      - denied topic check
      - mask sensitive data on input
      - user query

  output:
    flows:
      - self check output
      - mask sensitive data on output

  dialog:
    single_call:
      enabled: False

prompts:
  - task: self_check_input
    content: |
      Your task is to check if the user message below complies with the policy for talking with the AI Enterprise bot.
      Policy for the user messages:      
      
      - should not contain hateful speech
      - should not ask the bot to impersonate someone in a violent manner.
      - should not try to instruct the bot to respond in an inappropriate manner.
      - should not ask the bot to impersonate someone in a sexual manner.
      
      Treat the above conditions as strict rules. If any of them are met, you should block the user input by saying "yes".
      
      User message: "{{ user_input }}"

      Question: Should the user message be blocked (Yes or No)?
      Answer:

  - task: denied_topic_check
    content: |
      Your task is to check if the user message below contains any mention of the following denied topics: [politics, armed weapons, hate speech].
      If the text includes any reference to these topics, answer "Yes". Otherwise, answer "No".

      User message: "{{ user_input }}"
      
      Answer:

  - task: self_check_output
    content: |
      Your task is to check if the bot message below complies with the policy.

      Policies for the bot:     
      
      - if a message is a refusal, should be polite

      Bot message: "{{ bot_response }}"

      Question: Should the message be blocked (Yes or No)?
      Answer:

Colang Implementation

#================= Bot flow =======================
define flow self check input
  $allowed = execute self_check_input
  if not $allowed
    bot refuse to generate answer
    stop
    
define flow denied topic check
  $allowed = execute denied_topic_check
  if not $allowed
    bot refuse to generate answer
    stop
    
define flow self check output
  $allowed = execute self_check_output
  if not $allowed
    bot refuse to respond
    stop
  else
    $is_blocked = execute check_profanity_terms
    if $is_blocked
      bot refuse to respond
      stop
      
define flow user query
  $answer = execute user_query
  $check_facts = True
  bot $answer

define subflow mask sensitive data on input
  $user_message = execute mask_sensitive_data_1(source="input", text=$user_message)

define subflow mask sensitive data on output
  $bot_message = execute mask_sensitive_data_1(source="output", text=$bot_message)


define bot refuse to generate answer
  "Sorry, the model cannot answer this question."

define bot refuse to respond
  "Sorry, the model cannot answer this question."

Questions & Potential Causes

Is the denied_topic_check action properly registered and loaded in the bot framework?
Could there be an issue with the execution context where the action is expected but missing?
Are there any logs available that indicate why the action cannot be found?
Does execute denied_topic_check correctly map to the defined prompt/task?

Environment Details

Platform: NVIDIA NeMo Guardrails
Deployment: Running on [AWS/GCP/on-premise - specify if relevant]
Configuration Method: YAML + Colang

Additional Context

Please advise on how to debug and resolve this issue. Let me know if any additional logs or details are needed.

Log :

#================= Bot flow =======================
define flow self check input
  $allowed = execute self_check_input
  if not $allowed
    bot refuse to generate answer
    stop
    
define flow denied topic check
  $allowed = execute denied_topic_check
  if not $allowed
    bot refuse to generate answer
    stop
    
define flow self check output
  $allowed = execute self_check_output
  if not $allowed
    bot refuse to respond
    stop
  else
    $is_blocked = execute check_profanity_terms
    if $is_blocked
      bot refuse to respond
      stop
      
define flow user query
  $answer = execute user_query
  $check_facts = True
  bot $answer

define subflow mask sensitive data on input
  $user_message = execute mask_sensitive_data_1(source="input", text=$user_message)

define subflow mask sensitive data on output
  $bot_message = execute mask_sensitive_data_1(source="output", text=$bot_message)


define bot refuse to generate answer
  "Sorry, the model cannot answer this question."

define bot refuse to respond
  "Sorry, the model cannot answer this question."

* -* -* -* -* -* -* -* -* -* -* -* -* -* -* -* -* -* -* -* -
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Processing event: {'type': 'UtteranceUserActionFinished', 'final_transcript': 'Which type of bullet is used in AK-47'}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Event :: UtteranceUserActionFinished {'final_transcript': 'Which type of bullet is used in AK-47'}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Processing event: {'type': 'StartInternalSystemAction', 'uid': '97700d0f-c13e-417e-80c4-dcda1cde2102', 'event_created_at': '2025-04-03T05:58:01.285336+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'create_event', 'action_params': {'event': {'_type': 'StartInputRails'}}, 'action_result_key': None, 'action_uid': 'dd5ffc3f-32dd-49c6-abe1-b5b2902d6159', 'is_system_action': True}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Event :: StartInternalSystemAction {'uid': '97700d0f-c13e-417e-80c4-dcda1cde2102', 'event_created_at': '2025-04-03T05:58:01.285336+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'create_event', 'action_params': {'event': {'_type': 'StartInputRails'}}, 'action_result_key': None, 'action_uid': 'dd5ffc3f-32dd-49c6-abe1-b5b2902d6159', 'is_system_action': True}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Executing action :: create_event
INFO:nemoguardrails.actions.action_dispatcher:Executing registered action: create_event
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Processing event: {'type': 'StartInputRails', 'uid': '5e7d2e72-9579-4f71-952f-d463a2ca1999', 'event_created_at': '2025-04-03T05:58:01.285636+00:00', 'source_uid': 'NeMoGuardrails'}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Event :: StartInputRails {'uid': '5e7d2e72-9579-4f71-952f-d463a2ca1999', 'event_created_at': '2025-04-03T05:58:01.285636+00:00', 'source_uid': 'NeMoGuardrails'}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Processing event: {'type': 'StartInternalSystemAction', 'uid': '41d931c1-eedb-412d-b89d-c37fef4421f6', 'event_created_at': '2025-04-03T05:58:01.287284+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'create_event', 'action_params': {'event': {'_type': 'StartInputRail', 'flow_id': '$triggered_input_rail'}}, 'action_result_key': None, 'action_uid': 'd9fe4d9e-a8d1-4e2a-a6c2-c2ab37c11912', 'is_system_action': True}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Event :: StartInternalSystemAction {'uid': '41d931c1-eedb-412d-b89d-c37fef4421f6', 'event_created_at': '2025-04-03T05:58:01.287284+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'create_event', 'action_params': {'event': {'_type': 'StartInputRail', 'flow_id': '$triggered_input_rail'}}, 'action_result_key': None, 'action_uid': 'd9fe4d9e-a8d1-4e2a-a6c2-c2ab37c11912', 'is_system_action': True}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Executing action :: create_event
INFO:nemoguardrails.actions.action_dispatcher:Executing registered action: create_event
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Processing event: {'type': 'StartInputRail', 'uid': 'c1de429a-cb71-478f-b334-abbf9fb16494', 'event_created_at': '2025-04-03T05:58:01.287541+00:00', 'source_uid': 'NeMoGuardrails', 'flow_id': 'jailbreak detection heuristics'}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Event :: StartInputRail {'uid': 'c1de429a-cb71-478f-b334-abbf9fb16494', 'event_created_at': '2025-04-03T05:58:01.287541+00:00', 'source_uid': 'NeMoGuardrails', 'flow_id': 'jailbreak detection heuristics'}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Processing event: {'type': 'StartInternalSystemAction', 'uid': '8f60b7dd-4ea7-4aad-82ad-ffc277acd6be', 'event_created_at': '2025-04-03T05:58:01.288187+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'jailbreak_detection_heuristics', 'action_params': {}, 'action_result_key': 'is_jailbreak', 'action_uid': 'b06deb0e-9fc3-47b4-b07b-74e73d176820', 'is_system_action': False}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Event :: StartInternalSystemAction {'uid': '8f60b7dd-4ea7-4aad-82ad-ffc277acd6be', 'event_created_at': '2025-04-03T05:58:01.288187+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'jailbreak_detection_heuristics', 'action_params': {}, 'action_result_key': 'is_jailbreak', 'action_uid': 'b06deb0e-9fc3-47b4-b07b-74e73d176820', 'is_system_action': False}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Executing action :: jailbreak_detection_heuristics
INFO:nemoguardrails.actions.action_dispatcher:Executing registered action: jailbreak_detection_heuristics
WARNING:actions.py:No jailbreak detection endpoint set. Running in-process, NOT RECOMMENDED FOR PRODUCTION.
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Processing event: {'type': 'InternalSystemActionFinished', 'uid': 'af7304cd-4ef5-47e6-9c54-abb66d37569b', 'event_created_at': '2025-04-03T05:58:01.738422+00:00', 'source_uid': 'NeMoGuardrails', 'action_uid': 'b06deb0e-9fc3-47b4-b07b-74e73d176820', 'action_name': 'jailbreak_detection_heuristics', 'action_params': {}, 'action_result_key': 'is_jailbreak', 'status': 'success', 'is_success': True, 'return_value': False, 'events': [], 'is_system_action': False, 'action_finished_at': '2025-04-03T05:58:01.738434+00:00'}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Event :: InternalSystemActionFinished {'uid': 'af7304cd-4ef5-47e6-9c54-abb66d37569b', 'event_created_at': '2025-04-03T05:58:01.738422+00:00', 'source_uid': 'NeMoGuardrails', 'action_uid': 'b06deb0e-9fc3-47b4-b07b-74e73d176820', 'action_name': 'jailbreak_detection_heuristics', 'action_params': {}, 'action_result_key': 'is_jailbreak', 'status': 'success', 'is_success': True, 'return_value': False, 'events': [], 'is_system_action': False, 'action_finished_at': '2025-04-03T05:58:01.738434+00:00'}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Processing event: {'type': 'StartInternalSystemAction', 'uid': 'c21ad38f-8354-4402-9d86-7160dbe07b1a', 'event_created_at': '2025-04-03T05:58:01.739430+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'create_event', 'action_params': {'event': {'_type': 'InputRailFinished', 'flow_id': '$triggered_input_rail'}}, 'action_result_key': None, 'action_uid': 'c29c0e83-757e-4286-ad86-457edfc18072', 'is_system_action': True}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Event :: StartInternalSystemAction {'uid': 'c21ad38f-8354-4402-9d86-7160dbe07b1a', 'event_created_at': '2025-04-03T05:58:01.739430+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'create_event', 'action_params': {'event': {'_type': 'InputRailFinished', 'flow_id': '$triggered_input_rail'}}, 'action_result_key': None, 'action_uid': 'c29c0e83-757e-4286-ad86-457edfc18072', 'is_system_action': True}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Executing action :: create_event
INFO:nemoguardrails.actions.action_dispatcher:Executing registered action: create_event
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Processing event: {'type': 'InputRailFinished', 'uid': '9b5b0938-d0e8-4411-a823-8fb240672e1c', 'event_created_at': '2025-04-03T05:58:01.739763+00:00', 'source_uid': 'NeMoGuardrails', 'flow_id': 'jailbreak detection heuristics'}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Event :: InputRailFinished {'uid': '9b5b0938-d0e8-4411-a823-8fb240672e1c', 'event_created_at': '2025-04-03T05:58:01.739763+00:00', 'source_uid': 'NeMoGuardrails', 'flow_id': 'jailbreak detection heuristics'}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Processing event: {'type': 'StartInternalSystemAction', 'uid': 'b93dcdfb-9fac-4fb3-b497-237a058c2fd6', 'event_created_at': '2025-04-03T05:58:01.740916+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'create_event', 'action_params': {'event': {'_type': 'StartInputRail', 'flow_id': '$triggered_input_rail'}}, 'action_result_key': None, 'action_uid': 'c26c6e41-e061-43f6-9667-131012ec9195', 'is_system_action': True}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Event :: StartInternalSystemAction {'uid': 'b93dcdfb-9fac-4fb3-b497-237a058c2fd6', 'event_created_at': '2025-04-03T05:58:01.740916+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'create_event', 'action_params': {'event': {'_type': 'StartInputRail', 'flow_id': '$triggered_input_rail'}}, 'action_result_key': None, 'action_uid': 'c26c6e41-e061-43f6-9667-131012ec9195', 'is_system_action': True}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Executing action :: create_event
INFO:nemoguardrails.actions.action_dispatcher:Executing registered action: create_event
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Processing event: {'type': 'StartInputRail', 'uid': '19248016-55e7-48e6-b5cd-1c063f918018', 'event_created_at': '2025-04-03T05:58:01.741224+00:00', 'source_uid': 'NeMoGuardrails', 'flow_id': 'self check input'}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Event :: StartInputRail {'uid': '19248016-55e7-48e6-b5cd-1c063f918018', 'event_created_at': '2025-04-03T05:58:01.741224+00:00', 'source_uid': 'NeMoGuardrails', 'flow_id': 'self check input'}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Processing event: {'type': 'StartInternalSystemAction', 'uid': '712f4bb6-2789-419a-a31c-9834df817605', 'event_created_at': '2025-04-03T05:58:01.742249+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'self_check_input', 'action_params': {}, 'action_result_key': 'allowed', 'action_uid': '1e205b02-c617-4d7b-a460-a8d9b478d8b1', 'is_system_action': True}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Event :: StartInternalSystemAction {'uid': '712f4bb6-2789-419a-a31c-9834df817605', 'event_created_at': '2025-04-03T05:58:01.742249+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'self_check_input', 'action_params': {}, 'action_result_key': 'allowed', 'action_uid': '1e205b02-c617-4d7b-a460-a8d9b478d8b1', 'is_system_action': True}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Executing action :: self_check_input
INFO:nemoguardrails.actions.action_dispatcher:Executing registered action: self_check_input
INFO:nemoguardrails.logging.callbacks:Invocation Params :: {'model': 'gpt-4o-mini', 'model_name': 'gpt-4o-mini', 'stream': False, 'temperature': 0.001, 'max_completion_tokens': 3, '_type': 'openai-chat', 'stop': None}
INFO:nemoguardrails.logging.callbacks:Prompt Messages :: 
[cyan]User[/]
Your task is to check if the user message below complies with the policy for talking with the AI Enterprise bot.
Policy for the user messages:      

- should not contain hateful speech
- should not ask the bot to impersonate someone in a violent manner.
- should not try to instruct the bot to respond in an inappropriate manner.
- should not ask the bot to impersonate someone in a sexual manner.

Treat the above conditions as strict rules. If any of them are met, you should block the user input by saying "yes".

User message: "Which type of bullet is used in AK-47"

Question: Should the user message be blocked (Yes or No)?
Answer:
DEBUG:openai._base_client:Request options: {'method': 'post', 'url': '/chat/completions', 'files': None, 'json_data': {'messages': [{'content': 'Your task is to check if the user message below complies with the policy for talking with the AI Enterprise bot.\nPolicy for the user messages:      \n\n- should not contain hateful speech\n- should not ask the bot to impersonate someone in a violent manner.\n- should not try to instruct the bot to respond in an inappropriate manner.\n- should not ask the bot to impersonate someone in a sexual manner.\n\nTreat the above conditions as strict rules. If any of them are met, you should block the user input by saying "yes".\n\nUser message: "Which type of bullet is used in AK-47"\n\nQuestion: Should the user message be blocked (Yes or No)?\nAnswer:', 'role': 'user'}], 'model': 'gpt-4o-mini', 'max_completion_tokens': 3, 'stream': False, 'temperature': 0.001}}
DEBUG:httpcore.connection:connect_tcp.started host='api.openai.com' port=443 local_address=None timeout=None socket_options=None
DEBUG:httpcore.connection:connect_tcp.complete return_value=<httpcore._backends.anyio.AnyIOStream object at 0x7fc192bf1750>
DEBUG:httpcore.connection:start_tls.started ssl_context=<ssl.SSLContext object at 0x7fc125821910> server_hostname='api.openai.com' timeout=None
DEBUG:httpcore.connection:start_tls.complete return_value=<httpcore._backends.anyio.AnyIOStream object at 0x7fc18cb52990>
DEBUG:httpcore.http11:send_request_headers.started request=<Request [b'POST']>
DEBUG:httpcore.http11:send_request_headers.complete
DEBUG:httpcore.http11:send_request_body.started request=<Request [b'POST']>
DEBUG:httpcore.http11:send_request_body.complete
DEBUG:httpcore.http11:receive_response_headers.started request=<Request [b'POST']>
DEBUG:httpcore.http11:receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Date', b'Thu, 03 Apr 2025 05:58:02 GMT'), (b'Content-Type', b'application/json'), (b'Transfer-Encoding', b'chunked'), (b'Connection', b'keep-alive'), (b'access-control-expose-headers', b'X-Request-ID'), (b'openai-organization', b'user-dnp25mf3xbjlhii9rzrqsdc0'), (b'openai-processing-ms', b'246'), (b'openai-version', b'2020-10-01'), (b'x-ratelimit-limit-requests', b'10000'), (b'x-ratelimit-limit-tokens', b'10000000'), (b'x-ratelimit-remaining-requests', b'9999'), (b'x-ratelimit-remaining-tokens', b'9999838'), (b'x-ratelimit-reset-requests', b'6ms'), (b'x-ratelimit-reset-tokens', b'0s'), (b'x-request-id', b'req_f17d64f93cf931c0f76401b1893d14af'), (b'strict-transport-security', b'max-age=31536000; includeSubDomains; preload'), (b'cf-cache-status', b'DYNAMIC'), (b'Set-Cookie', b'__cf_bm=9QN7KEaqRLTlGsCI_ry0_P3D4WKl58AWWjNIQ05.z9c-1743659882-1.0.1.1-zkgPBAXwQgaSWX9Lx6yXzSKYgQOW4gd2WaJn5vB6u7gLCr6p.NxCNMxp3377McGMcSkwmNt124cv5YEJ8Z_cPBiX7n6_f6YsRgqV18KGHKU; path=/; expires=Thu, 03-Apr-25 06:28:02 GMT; domain=.api.openai.com; HttpOnly; Secure; SameSite=None'), (b'X-Content-Type-Options', b'nosniff'), (b'Set-Cookie', b'_cfuvid=w8Ywz.8OZMFL3o5zz3F1UIMlz2XPk384Um1FkYYkJzc-1743659882053-0.0.1.1-604800000; path=/; domain=.api.openai.com; HttpOnly; Secure; SameSite=None'), (b'Server', b'cloudflare'), (b'CF-RAY', b'92a654f50b49c599-IAD'), (b'Content-Encoding', b'gzip'), (b'alt-svc', b'h3=":443"; ma=86400')])
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
DEBUG:httpcore.http11:receive_response_body.started request=<Request [b'POST']>
DEBUG:httpcore.http11:receive_response_body.complete
DEBUG:httpcore.http11:response_closed.started
DEBUG:httpcore.http11:response_closed.complete
DEBUG:openai._base_client:HTTP Request: POST https://api.openai.com/v1/chat/completions "200 OK"
INFO:nemoguardrails.logging.callbacks:Completion :: No
INFO:nemoguardrails.logging.callbacks:Output Stats :: {'token_usage': {'completion_tokens': 2, 'prompt_tokens': 144, 'total_tokens': 146, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}, 'model_name': 'gpt-4o-mini-2024-07-18', 'system_fingerprint': 'fp_86d0290411', 'id': 'chatcmpl-BI7sPnwWpRk57qKizf7qkAiuYImJM'}
INFO:nemoguardrails.logging.callbacks:--- :: LLM call took 0.31 seconds
INFO:actions.py:Input self-checking result is: `No`.
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Processing event: {'type': 'InternalSystemActionFinished', 'uid': 'ea25ea72-5346-4fe3-a8a1-c891000dbac1', 'event_created_at': '2025-04-03T05:58:02.059075+00:00', 'source_uid': 'NeMoGuardrails', 'action_uid': '1e205b02-c617-4d7b-a460-a8d9b478d8b1', 'action_name': 'self_check_input', 'action_params': {}, 'action_result_key': 'allowed', 'status': 'success', 'is_success': True, 'return_value': True, 'events': [], 'is_system_action': True, 'action_finished_at': '2025-04-03T05:58:02.059084+00:00'}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Event :: InternalSystemActionFinished {'uid': 'ea25ea72-5346-4fe3-a8a1-c891000dbac1', 'event_created_at': '2025-04-03T05:58:02.059075+00:00', 'source_uid': 'NeMoGuardrails', 'action_uid': '1e205b02-c617-4d7b-a460-a8d9b478d8b1', 'action_name': 'self_check_input', 'action_params': {}, 'action_result_key': 'allowed', 'status': 'success', 'is_success': True, 'return_value': True, 'events': [], 'is_system_action': True, 'action_finished_at': '2025-04-03T05:58:02.059084+00:00'}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Processing event: {'type': 'StartInternalSystemAction', 'uid': '6ba5fbbd-989d-47c0-a869-7267241c09e9', 'event_created_at': '2025-04-03T05:58:02.060111+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'create_event', 'action_params': {'event': {'_type': 'InputRailFinished', 'flow_id': '$triggered_input_rail'}}, 'action_result_key': None, 'action_uid': 'd4f52ee6-778b-4106-9d6a-b19c323ffe4a', 'is_system_action': True}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Event :: StartInternalSystemAction {'uid': '6ba5fbbd-989d-47c0-a869-7267241c09e9', 'event_created_at': '2025-04-03T05:58:02.060111+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'create_event', 'action_params': {'event': {'_type': 'InputRailFinished', 'flow_id': '$triggered_input_rail'}}, 'action_result_key': None, 'action_uid': 'd4f52ee6-778b-4106-9d6a-b19c323ffe4a', 'is_system_action': True}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Executing action :: create_event
INFO:nemoguardrails.actions.action_dispatcher:Executing registered action: create_event
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Processing event: {'type': 'InputRailFinished', 'uid': '5d497d12-10d5-41b9-b415-61cc944afeab', 'event_created_at': '2025-04-03T05:58:02.060379+00:00', 'source_uid': 'NeMoGuardrails', 'flow_id': 'self check input'}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Event :: InputRailFinished {'uid': '5d497d12-10d5-41b9-b415-61cc944afeab', 'event_created_at': '2025-04-03T05:58:02.060379+00:00', 'source_uid': 'NeMoGuardrails', 'flow_id': 'self check input'}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Processing event: {'type': 'StartInternalSystemAction', 'uid': 'ec48d63a-df00-476d-a8ea-bf55058ec088', 'event_created_at': '2025-04-03T05:58:02.061680+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'create_event', 'action_params': {'event': {'_type': 'StartInputRail', 'flow_id': '$triggered_input_rail'}}, 'action_result_key': None, 'action_uid': '6480f39a-f552-4244-a8f3-22535cd18658', 'is_system_action': True}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Event :: StartInternalSystemAction {'uid': 'ec48d63a-df00-476d-a8ea-bf55058ec088', 'event_created_at': '2025-04-03T05:58:02.061680+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'create_event', 'action_params': {'event': {'_type': 'StartInputRail', 'flow_id': '$triggered_input_rail'}}, 'action_result_key': None, 'action_uid': '6480f39a-f552-4244-a8f3-22535cd18658', 'is_system_action': True}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Executing action :: create_event
INFO:nemoguardrails.actions.action_dispatcher:Executing registered action: create_event
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Processing event: {'type': 'StartInputRail', 'uid': '8176d554-199e-48df-966c-d196854d5d5f', 'event_created_at': '2025-04-03T05:58:02.061943+00:00', 'source_uid': 'NeMoGuardrails', 'flow_id': 'denied topic check'}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Event :: StartInputRail {'uid': '8176d554-199e-48df-966c-d196854d5d5f', 'event_created_at': '2025-04-03T05:58:02.061943+00:00', 'source_uid': 'NeMoGuardrails', 'flow_id': 'denied topic check'}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Processing event: {'type': 'StartInternalSystemAction', 'uid': 'd0afdf25-9da9-4306-a166-a50fe0a207c4', 'event_created_at': '2025-04-03T05:58:02.063296+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'denied_topic_check', 'action_params': {}, 'action_result_key': 'allowed', 'action_uid': 'c5959484-ff57-4890-8aa1-bfe00411124e', 'is_system_action': False}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Event :: StartInternalSystemAction {'uid': 'd0afdf25-9da9-4306-a166-a50fe0a207c4', 'event_created_at': '2025-04-03T05:58:02.063296+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'denied_topic_check', 'action_params': {}, 'action_result_key': 'allowed', 'action_uid': 'c5959484-ff57-4890-8aa1-bfe00411124e', 'is_system_action': False}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Processing event: {'type': 'hide_prev_turn'}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Event :: hide_prev_turn {}
INFO:nemoguardrails.rails.llm.llmrails:--- :: Total processing took 0.78 seconds. LLM Stats: 1 total calls, 0.31 total time, 146 total tokens, 144 total prompt tokens, 2 total completion tokens, [0.31] as latencies
* - * - * - * - * - * - * - * - * - * - 
({'role': 'assistant', 'content': "Action 'denied_topic_check' not found."}, {'last_user_message': None, 'last_bot_message': "Action 'denied_topic_check' not found.", 'generation_options': {'rails': {'input': True, 'output': True, 'retrieval': True, 'dialog': True}, 'llm_params': None, 'llm_output': False, 'output_vars': True, 'log': {'activated_rails': False, 'llm_calls': False, 'internal_events': False, 'colang_history': False}}, 'user_message': 'Which type of bullet is used in AK-47', 'i': 2, 'input_flows': ['jailbreak detection heuristics', 'self check input', 'denied topic check', 'mask sensitive data on input', 'user query'], 'triggered_input_rail': 'denied topic check', 'is_jailbreak': False, 'allowed': True, 'event': {'type': 'Listen', 'uid': 'cb39a4e3-2b19-45c7-9e2a-75473744620d', 'event_created_at': '2025-04-03T05:58:02.063576+00:00', 'source_uid': 'NeMoGuardrails'}})

### Steps To Reproduce

YML content :
```yml
models:
  - type: main
    engine: openai
    model: gpt-4o-mini

instructions:
  - type: general
    content: |
      You are a helpful assistant that can answer given questions. Bot should not go off-topic.

rails:
  config: 
    sensitive_data_detection:
      input:
        entities:
          - IN_PAN
          - IN_PASSPORT
          - PHONE_NUMBER
          - EMAIL_ADDRESS
          - DATE_TIME
      output:
        entities:
          - IN_PAN
          - IN_PASSPORT
          - PHONE_NUMBER
          - EMAIL_ADDRESS
          - DATE_TIME

  input:
    flows:
      - jailbreak detection heuristics
      - self check input
      - denied topic check
      - mask sensitive data on input
      - user query

  output:
    flows:
      - self check output
      - mask sensitive data on output

  dialog:
    single_call:
      enabled: False

prompts:
  - task: self_check_input
    content: |
      Your task is to check if the user message below complies with the policy for talking with the AI Enterprise bot.
      Policy for the user messages:      
      
      - should not contain hateful speech
      - should not ask the bot to impersonate someone in a violent manner.
      - should not try to instruct the bot to respond in an inappropriate manner.
      - should not ask the bot to impersonate someone in a sexual manner.
      
      Treat the above conditions as strict rules. If any of them are met, you should block the user input by saying "yes".
      
      User message: "{{ user_input }}"

      Question: Should the user message be blocked (Yes or No)?
      Answer:

  - task: denied_topic_check
    content: |
      Your task is to check if the user message below contains any mention of the following denied topics: [politics, armed weapons, hate speech].
      If the text includes any reference to these topics, answer "Yes". Otherwise, answer "No".

      User message: "{{ user_input }}"
      
      Answer:

  - task: self_check_output
    content: |
      Your task is to check if the bot message below complies with the policy.

      Policies for the bot:     
      
      - if a message is a refusal, should be polite

      Bot message: "{{ bot_response }}"

      Question: Should the message be blocked (Yes or No)?
      Answer:

Colang content :

#================= Bot flow =======================
define flow self check input
  $allowed = execute self_check_input
  if not $allowed
    bot refuse to generate answer
    stop
    
define flow denied topic check
  $allowed = execute denied_topic_check
  if not $allowed
    bot refuse to generate answer
    stop
    
define flow self check output
  $allowed = execute self_check_output
  if not $allowed
    bot refuse to respond
    stop
  else
    $is_blocked = execute check_profanity_terms
    if $is_blocked
      bot refuse to respond
      stop
      
define flow user query
  $answer = execute user_query
  $check_facts = True
  bot $answer

define subflow mask sensitive data on input
  $user_message = execute mask_sensitive_data_1(source="input", text=$user_message)

define subflow mask sensitive data on output
  $bot_message = execute mask_sensitive_data_1(source="output", text=$bot_message)


define bot refuse to generate answer
  "Sorry, the model cannot answer this question."

define bot refuse to respond
  "Sorry, the model cannot answer this question."

Expected Behavior

It should execute denied topic not to say Action 'denied_topic_check' not found.

Actual Behavior

It should have blocked the message and say : Sorry, the model cannot answer this question.

Apr 03 '25 06:04 mohilmakwana31

NeMo-Guardrails NeMo-Guardrails copied to clipboard

bug: Bot fails to execute 'denied_topic_check' action despite defined YAML and Colang flows

Did you check docs and existing issues?

Python version (python --version)

Operating system/version

NeMo-Guardrails version (if you must use a specific version and not the latest

Describe the bug

Steps to Reproduce

Expected Behavior

Actual Behavior

Relevant Configuration Details

YAML (rails section)

Colang Implementation

Questions & Potential Causes

Environment Details

Additional Context

Expected Behavior

Actual Behavior

NeMo-Guardrails
NeMo-Guardrails copied to clipboard

YAML (`rails` section)