Allow handling LLM generation exceptions

[drazvan]

If a third party API is used, some external guardrails can block the LLM generation. If this happens, currently a None response is returned. There should be an option to gracefully handle these types of errors.

[lexkornyukhin]

Hi, any updates on this one? I am experiencing an issue when using Azure OpenAI as the LLM for guardrail. Due to their internal filtering an exception is raised when you send harmful content (e.g. speech), but it is impossible to get this exception back from NeMo-Guardrails object. Getting the text in console though:

Error Error code: 400 - {'error': {'message': "The response was filtered due to the prompt triggering Azure OpenAI's content management policy. Please modify your prompt and retry. To learn more about our content filtering policies please read our documentation: https://go.microsoft.com/fwlink/?linkid=2198766", 'type': None, 'param': 'prompt', 'code': 'content_filter', 'status': 400, 'innererror': {'code': 'ResponsibleAIPolicyViolation', 'content_filter_result': {'hate': {'filtered': True, 'severity': 'medium'}, 'self_harm': {'filtered': False, 'severity': 'safe'}, 'sexual': {'filtered': False, 'severity': 'safe'}, 'violence': {'filtered': False, 'severity': 'safe'}}}}} while execution self_check_input Traceback (most recent call last): File "/Users/alexander.kornyukhin/opt/anaconda3/envs/python3.11/lib/python3.11/site-packages/nemoguardrails/actions/action_dispatcher.py", line 178, in execute_action result = await result ^^^^^^^^^^^^ File "/Users/alexander.kornyukhin/opt/anaconda3/envs/python3.11/lib/python3.11/site-packages/nemoguardrails/actions/../library/self_check/input_check/actions.py", line 62, in self_check_input check = await llm_call(llm, prompt) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/alexander.kornyukhin/opt/anaconda3/envs/python3.11/lib/python3.11/site-packages/nemoguardrails/actions/llm/utils.py", line 53, in llm_call result = await llm.agenerate_prompt( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/alexander.kornyukhin/opt/anaconda3/envs/python3.11/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py", line 554, in agenerate_prompt return await self.agenerate( ^^^^^^^^^^^^^^^^^^^^^ File "/Users/alexander.kornyukhin/opt/anaconda3/envs/python3.11/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py", line 514, in agenerate raise exceptions[0] File "/Users/alexander.kornyukhin/opt/anaconda3/envs/python3.11/lib/python3.11/asyncio/tasks.py", line 267, in __step result = coro.send(None) ^^^^^^^^^^^^^^^ File "/Users/alexander.kornyukhin/opt/anaconda3/envs/python3.11/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py", line 617, in _agenerate_with_cache return await self._agenerate( ^^^^^^^^^^^^^^^^^^^^^^ File "/Users/alexander.kornyukhin/opt/anaconda3/envs/python3.11/lib/python3.11/site-packages/langchain_openai/chat_models/base.py", line 546, in _agenerate response = await self.async_client.create(messages=message_dicts, **params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ... ^^^^^^^^^^^^^^^^^^^^ File "/Users/alexander.kornyukhin/opt/anaconda3/envs/python3.11/lib/python3.11/site-packages/openai/_base_client.py", line 1499, in _request raise self._make_status_error_from_response(err.response) from None openai.BadRequestError: Error code: 400 - {'error': {'message': "The response was filtered due to the prompt triggering Azure OpenAI's content management policy. Please modify your prompt and retry. To learn more about our content filtering policies please read our documentation: https://go.microsoft.com/fwlink/?linkid=2198766", 'type': None, 'param': 'prompt', 'code': 'content_filter', 'status': 400, 'innererror': {'code': 'ResponsibleAIPolicyViolation', 'content_filter_result': {'hate': {'filtered': True, 'severity': 'medium'}, 'self_harm': {'filtered': False, 'severity': 'safe'}, 'sexual': {'filtered': False, 'severity': 'safe'}, 'violence': {'filtered': False, 'severity': 'safe'}}}}}

[drazvan]

Not yet. We might for the next release. In 0.8.0 which will be published tomorrow we've added support for retrieving data out of the context: https://github.com/NVIDIA/NeMo-Guardrails/blob/develop/docs/user_guides/advanced/generation-options.md#output-variables

To use this, as a temporary workaround, you can override the self_check_input action, add a try block and save the exception in a context variable. Something along the lines:

@action(is_system_action=True)
async def self_check_input(
    llm_task_manager: LLMTaskManager,
    context: Optional[dict] = None,
    llm: Optional[BaseLLM] = None,
):
    """Checks the input from the user.

    Prompt the LLM, using the `check_input` task prompt, to determine if the input
    from the user should be allowed or not.

    Returns:
        True if the input should be allowed, False otherwise.
    """

    user_input = context.get("user_message")

    if user_input:
        prompt = llm_task_manager.render_task_prompt(
            task=Task.SELF_CHECK_INPUT,
            context={
                "user_input": user_input,
            },
        )
        stop = llm_task_manager.get_stop_tokens(task=Task.SELF_CHECK_INPUT)

        # Initialize the LLMCallInfo object
        llm_call_info_var.set(LLMCallInfo(task=Task.SELF_CHECK_INPUT.value))

+        context_updates = {
+            "llm_exception": None
+        }
+        try:
            with llm_params(llm, temperature=0.0):
                check = await llm_call(llm, prompt, stop=stop)
+        except Exception as ex:
+            # Here you can parse the data from the exception instead of str(ex)
+            context_updates["llm_exception"] = str(ex)

        check = check.lower().strip()
        log.info(f"Input self-checking result is: `{check}`.")

        if "yes" in check:
            return ActionResult(
                return_value=False,
                events=[
                    new_event_dict(
                        "mask_prev_user_message", intent="unanswerable message"
                    )
                ],
+              context_updates=context_updates
            )

    return True

Let me know if this is helpful.

[lexkornyukhin]

Updated the lib to the latest version and was following this guide, but encountered: TypeError: LLMRails.generate() got an unexpected keyword argument 'options'

Also, should I expect the same behaviour with RunnableRails or this is something which is currently implemented in LLMRails?

[drazvan]

@lexkornyukhin : The first error is likely because you have to pip install -e . again, i.e., you've updated the code, but not reinstalled.

You can also try the changes here: #384 (it's WIP). If you return an event whose type ends with "Exception" it would be propagated as the response.

Let me know if this helps.