Initial slot values break rule policy by being integrated into all rules by default

[twerkmeister]

Rasa Open Source version

3.2.4

Rasa SDK version

3.2.0

Python version

3.7

What operating system are you using?

Linux, OSX

What happened?

Was trying to build a concept bot with some basic task switching logic and noticed I couldn't get certain rules to work. After some bug hunting I found that initial slot values are implicitly integrated into all rules. Thus, if you change the initial value during the conversation, none of your rules will apply anymore: 🥳🎉

Minimal example to reproduce:

nlu

version: "3.1"

nlu:
- intent: greet
  examples: |
    - hey
    - hello
    - hi
    - hello there
    - good morning
    - good evening
    - moin
    - hey there
    - let's go
    - hey dude
    - goodmorning
    - goodevening
    - good afternoon

- intent: goodbye
  examples: |
    - cu
    - good by
    - cee you later
    - good night
    - bye
    - goodbye
    - have a nice day
    - see you around
    - bye bye
    - see you later

rules

version: "3.1"

rules:

- rule: Say hi!
  steps:
    - intent: greet
    - action: utter_greet

- rule: Say goodbye anytime the user says goodbye
  steps:
  - intent: goodbye
  - action: utter_goodbye

config: default

domain

version: "3.1"

intents:
  - greet
  - goodbye

slots:
  has_said_hi:
    type: bool
    initial_value: false # commenting this out will make the rules work again
    mappings:
      - type: from_intent
        value: true
        intent: greet

responses:
  utter_greet:
  - text: "Hey! How are you?"

  utter_goodbye:
  - text: "Bye"

session_config:
  session_expiration_time: 60
  carry_over_slots_to_new_session: true

When you have a look at the rules that are produced with and without initial value for the slot has_said_hi you will see the following:

rules in the trained rule_policy.json (from the model tar.gz) without initial value:

    "rules": {
      "[{\"prev_action\": {\"action_name\": \"action_listen\"}, \"user\": {\"intent\": \"goodbye\"}}]": "utter_goodbye",
      "[{\"prev_action\": {\"action_name\": \"action_listen\"}, \"user\": {\"intent\": \"goodbye\"}}, {\"prev_action\": {\"action_name\": \"utter_goodbye\"}, \"user\": {\"intent\": \"goodbye\"}}]": "action_listen",
      "[{\"prev_action\": {\"action_name\": \"action_listen\"}, \"user\": {\"intent\": \"greet\"}}]": "utter_greet",
      "[{\"prev_action\": {\"action_name\": \"action_listen\"}, \"user\": {\"intent\": \"greet\"}}, {\"prev_action\": {\"action_name\": \"utter_greet\"}, \"user\": {\"intent\": \"greet\"}}]": "action_listen"
    },

rules in the trained rule_policy.json (from the model tar.gz) with initial value:

    "rules": {
      "[{\"prev_action\": {\"action_name\": \"action_listen\"}, \"slots\": {\"has_said_hi\": [1.0, 0.0]}, \"user\": {\"intent\": \"goodbye\"}}]": "utter_goodbye",
      "[{\"prev_action\": {\"action_name\": \"action_listen\"}, \"slots\": {\"has_said_hi\": [1.0, 0.0]}, \"user\": {\"intent\": \"goodbye\"}}, {\"prev_action\": {\"action_name\": \"utter_goodbye\"}, \"slots\": {\"has_said_hi\": [1.0, 0.0]}, \"user\": {\"intent\": \"goodbye\"}}]": "action_listen",
      "[{\"prev_action\": {\"action_name\": \"action_listen\"}, \"slots\": {\"has_said_hi\": [1.0, 0.0]}, \"user\": {\"intent\": \"greet\"}}]": "utter_greet",
      "[{\"prev_action\": {\"action_name\": \"action_listen\"}, \"slots\": {\"has_said_hi\": [1.0, 0.0]}, \"user\": {\"intent\": \"greet\"}}, {\"prev_action\": {\"action_name\": \"utter_greet\"}, \"slots\": {\"has_said_hi\": [1.0, 0.0]}, \"user\": {\"intent\": \"greet\"}}]": "action_listen"
    }

As you can see the initial value for has_said_hi was incorporated into all rules as [1.0, 0.0] which is the boolean slot representation for false.

What you will experience when you train this bot is the following: the greet rule won't work. What happens is that the slot has_said_hi get's filled, and afterwards there are no applicable rules and the bot will reset the previous user message (unsetting the slot again). So you can get a good bye but never get the bot to greet you back.

I think it's quite a serious bug, as it basically renders rules almost useless when using slots with initial values and it is super surprising that initial values are causing this. Instead, initial values should not be incorporated into rules.

Command / Request

`rasa train` with the provided example config

`rasa shell -vv` to interact with the bot and see debug info.

Relevant log output

Bot loaded. Type a message and press enter (use '/stop' to exit):
Your input ->  hi
2022-07-28 10:27:30 DEBUG    rasa.core.lock_store  - Issuing ticket for conversation 'a4fa8fc9e981468499d002ae25a30667'.
2022-07-28 10:27:30 DEBUG    rasa.core.lock_store  - Acquiring lock for conversation 'a4fa8fc9e981468499d002ae25a30667'.
2022-07-28 10:27:30 DEBUG    rasa.core.lock_store  - Acquired lock for conversation 'a4fa8fc9e981468499d002ae25a30667'.
2022-07-28 10:27:30 DEBUG    rasa.core.tracker_store  - Could not find tracker for conversation ID 'a4fa8fc9e981468499d002ae25a30667'.
2022-07-28 10:27:30 DEBUG    rasa.core.processor  - Starting a new session for conversation ID 'a4fa8fc9e981468499d002ae25a30667'.
2022-07-28 10:27:30 DEBUG    rasa.core.processor  - Policy prediction ended with events '[]'.
2022-07-28 10:27:30 DEBUG    rasa.core.processor  - Action 'action_session_start' ended with events '[<rasa.shared.core.events.SessionStarted object at 0x7f572869afd0>, ActionExecuted(action: action_listen, policy:
 None, confidence: None)]'.
2022-07-28 10:27:30 DEBUG    rasa.core.processor  - Current slot values:
        has_said_hi: False
        session_started_metadata: None
2022-07-28 10:27:30 DEBUG    rasa.engine.runner.dask  - Running graph with inputs: {'__message__': [<rasa.core.channels.channel.UserMessage object at 0x7f56985cb210>]}, targets: ['run_RegexMessageHandler'] and ExecutionContext(model_id='8923f28deae949efb9c00d8482a594b5', should_add_diagnostic_data=False, is_finetuning=False, node_name=None).
2022-07-28 10:27:30 DEBUG    rasa.engine.graph  - Node 'nlu_message_converter' running 'NLUMessageConverter.convert_user_message'.
2022-07-28 10:27:30 DEBUG    rasa.engine.graph  - Node 'run_WhitespaceTokenizer0' running 'WhitespaceTokenizer.process'.
2022-07-28 10:27:30 DEBUG    rasa.engine.graph  - Node 'run_RegexFeaturizer1' running 'RegexFeaturizer.process'.
2022-07-28 10:27:30 DEBUG    rasa.engine.graph  - Node 'run_LexicalSyntacticFeaturizer2' running 'LexicalSyntacticFeaturizer.process'.
2022-07-28 10:27:30 DEBUG    rasa.engine.graph  - Node 'run_CountVectorsFeaturizer3' running 'CountVectorsFeaturizer.process'.
2022-07-28 10:27:30 DEBUG    rasa.engine.graph  - Node 'run_CountVectorsFeaturizer4' running 'CountVectorsFeaturizer.process'.
2022-07-28 10:27:30 DEBUG    rasa.engine.graph  - Node 'run_DIETClassifier5' running 'DIETClassifier.process'.
2022-07-28 10:27:30 DEBUG    rasa.engine.graph  - Node 'run_EntitySynonymMapper6' running 'EntitySynonymMapper.process'.
2022-07-28 10:27:30 DEBUG    rasa.engine.graph  - Node 'run_ResponseSelector7' running 'ResponseSelector.process'.
2022-07-28 10:27:30 DEBUG    rasa.nlu.classifiers.diet_classifier  - There is no trained model for 'ResponseSelector': The component is either not trained or didn't receive enough training data.
2022-07-28 10:27:30 DEBUG    rasa.nlu.selectors.response_selector  - Adding following selector key to message property: default
2022-07-28 10:27:30 DEBUG    rasa.engine.graph  - Node 'run_FallbackClassifier8' running 'FallbackClassifier.process'.
2022-07-28 10:27:30 DEBUG    rasa.engine.graph  - Node 'domain_provider' running 'DomainProvider.provide_inference'.
2022-07-28 10:27:30 DEBUG    rasa.engine.graph  - Node 'run_RegexMessageHandler' running 'RegexMessageHandler.process'.
2022-07-28 10:27:30 DEBUG    rasa.core.processor  - Received user message 'hi' with intent '{'name': 'greet', 'confidence': 1.0}' and entities '[]'
2022-07-28 10:27:30 DEBUG    rasa.core.processor  - Logged UserUtterance - tracker now has 4 events.
2022-07-28 10:27:30 DEBUG    rasa.core.actions.action  - Validating extracted slots: has_said_hi
2022-07-28 10:27:30 DEBUG    rasa.core.processor  - Default action 'action_extract_slots' was executed, resulting in 1 events: SlotSet(key: has_said_hi, value: True)
2022-07-28 10:27:30 DEBUG    rasa.engine.runner.dask  - Running graph with inputs: {'__tracker__': <rasa.shared.core.trackers.DialogueStateTracker object at 0x7f56c80b69d0>}, targets: ['select_prediction'] and ExecutionContext(model_id='8923f28deae949efb9c00d8482a594b5', should_add_diagnostic_data=False, is_finetuning=False, node_name=None).
2022-07-28 10:27:30 DEBUG    rasa.engine.graph  - Node 'rule_only_data_provider' running 'RuleOnlyDataProvider.provide'.
2022-07-28 10:27:30 DEBUG    rasa.engine.graph  - Node 'domain_provider' running 'DomainProvider.provide_inference'.
2022-07-28 10:27:30 DEBUG    rasa.engine.graph  - Node 'run_MemoizationPolicy0' running 'MemoizationPolicy.predict_action_probabilities'.
2022-07-28 10:27:30 DEBUG    rasa.core.policies.memoization  - Current tracker state:
[state 1] user intent: greet | previous action name: action_listen
2022-07-28 10:27:30 DEBUG    rasa.core.policies.memoization  - There is no memorised next action
2022-07-28 10:27:30 DEBUG    rasa.engine.graph  - Node 'run_RulePolicy1' running 'RulePolicy.predict_action_probabilities'.
2022-07-28 10:27:30 DEBUG    rasa.core.policies.rule_policy  - Current tracker state:
[state 0] slots: {'has_said_hi': (1.0, 0.0)}
[state 1] user text: hi | previous action name: action_listen | slots: {'has_said_hi': (1.0, 1.0)}
2022-07-28 10:27:30 DEBUG    rasa.core.policies.rule_policy  - There is no applicable rule.
2022-07-28 10:27:30 DEBUG    rasa.core.policies.rule_policy  - Current tracker state:
[state 0] slots: {'has_said_hi': (1.0, 0.0)}
[state 1] user intent: greet | previous action name: action_listen | slots: {'has_said_hi': (1.0, 1.0)}
2022-07-28 10:27:30 DEBUG    rasa.core.policies.rule_policy  - There is no applicable rule.
2022-07-28 10:27:30 DEBUG    rasa.engine.graph  - Node 'run_TEDPolicy3' running 'TEDPolicy.predict_action_probabilities'.
2022-07-28 10:27:30 DEBUG    rasa.engine.graph  - Node 'run_UnexpecTEDIntentPolicy2' running 'UnexpecTEDIntentPolicy.predict_action_probabilities'.
2022-07-28 10:27:30 DEBUG    rasa.engine.graph  - Node 'select_prediction' running 'DefaultPolicyPredictionEnsemble.combine_predictions_from_kwargs'.
2022-07-28 10:27:30 DEBUG    rasa.core.policies.ensemble  - Made prediction using user intent.
2022-07-28 10:27:30 DEBUG    rasa.core.policies.ensemble  - Added `DefinePrevUserUtteredFeaturization(False)` event.
2022-07-28 10:27:30 DEBUG    rasa.core.policies.ensemble  - Predicted next action using RulePolicy.
2022-07-28 10:27:30 DEBUG    rasa.core.processor  - Predicted next action 'action_default_fallback' with confidence 0.30.
2022-07-28 10:27:30 DEBUG    rasa.core.processor  - Policy prediction ended with events '[<rasa.shared.core.events.DefinePrevUserUtteredFeaturization object at 0x7f56982be350>]'.
2022-07-28 10:27:30 DEBUG    rasa.core.processor  - Action 'action_default_fallback' ended with events '[<rasa.shared.core.events.UserUtteranceReverted object at 0x7f56985a1410>]'.
2022-07-28 10:27:30 DEBUG    rasa.core.processor  - Current slot values:
        has_said_hi: False
        session_started_metadata: None
2022-07-28 10:27:30 DEBUG    rasa.core.processor  - Predicted next action 'action_listen' with confidence 1.00.
2022-07-28 10:27:30 DEBUG    rasa.core.processor  - Policy prediction ended with events '[]'.
2022-07-28 10:27:30 DEBUG    rasa.core.processor  - Action 'action_listen' ended with events '[]'.
2022-07-28 10:27:30 DEBUG    rasa.core.lock_store  - Deleted lock for conversation 'a4fa8fc9e981468499d002ae25a30667'.
Your input ->