bug: Bot Producing Random and Incorrect Responses ("Ol", "O", "A") - Issue Report

[grupocopa]

Did you check docs and existing issues?

• [x] I have read all the NeMo-Guardrails docs
• [x] I have updated the package to the latest version before submitting this issue
• [ ] (optional) I have used the develop branch
• [x] I have searched the existing issues of NeMo-Guardrails

Python version (python --version)

Python 3.10.11

Operating system/version

Windows 11

NeMo-Guardrails version (if you must use a specific version and not the latest

0.11.0

Describe the bug

nemo_guardrails_a_error.txt intents_inside_awnser_error.txt nemo_guardrails_a_error.txt Subject: Bot Producing Random and Incorrect Responses ("Ol", "O", "A") - Issue Report

Hello NVIDIA Developer Community,

We are using the NVIDIA NeMo Guardrails framework in a system where a user-facing product interacts with a bot hosted via HTTP requests on a separate backend. While the bot operates as expected initially, it eventually starts producing random, incorrect responses such as "Ol", "O", or completely blank outputs.

This issue persists until we reset the hosting server and thread ID, which temporarily restores normal behavior. However, the need for frequent resets significantly impacts the reliability of our system. When the system resets, all goes well for a short period of time, and the problems surges again and persists until the next reset.

Example Behavior

At an undefined point during operation, instead of returning coherent responses to user inputs, the bot begins responding with seemingly random or nonsensical outputs like:

• "Ol"
• "O"
• Empty or blank responses
• Repeated system-level warnings and unrelated logs
• Sending literally intents to user

Details

• The logs for the affected sessions will be attached to this post for clarity and further investigation.
• The issue occurs consistently after the bot has been running for some time, though the exact trigger remains unclear.

Attached Logs

For better understanding, we are attaching a selection of logs captured during sessions where these issues occurred. The attached logs highlight examples of:

1. Incorrect or nonsensical responses.
2. Warnings and anomalies recorded by the system.

We would greatly appreciate any insights or suggestions from the community regarding the cause and potential resolution for this issue.
nemo_guardrails_a_error.txt intents_inside_awnser_error.txt

Thank you for your support!

Steps To Reproduce

I don't know. The server is going working well in 1-2 days, and then start to have those problems and errors.

Expected Behavior

The server completely normal like when it started.

Actual Behavior

Bot awnsering random, incorrect or blank responses.

[Pouyanpi]

@grupocopa thanks for sharing this, I think it really depends on the LLM you are using, I can imagine when the conversation history gets very long you might face issues ( "The server is going working well in 1-2 days, and then start to have those problems and errors." can confirm it).

So, the server's history management is primitive in a sense that it only appends new messages to the conversation history and use it in the subsequent turns of the conversation, and as the history gets larger the LLM's capacity to generate proper response diminishes. I cannot think of a WAR for this at the moment unless you are able to implement your own server that uses nemoguardrails package. We appreciate any contribution that could potentially improve it for the community.

For production use cases you can use NeMo Guardrails Microservice as part of NeMo Platform in near future.