Training problem with LanguageModelFeaturizer in Rasa 3

[navidved]

Rasa Open Source version

3.2.5

Rasa SDK version

3.2.0

Python version

3.8

What operating system are you using?

Windows

What happened?

Hey, I download fresh Rasa and use default Data (nlu.yml, rules , ... ) and try to train it with LanguageModelFeaturizer and BERT , I didn't change anything except the config. I changed it according to documentation I am getting this error while training the model : raise GraphComponentException( rasa.engine.exceptions.GraphComponentException: Error initializing graph component for node run_LanguageModelFeaturizer1.

rasa --version

Rasa Version : 3.2.5 Minimum Compatible Version: 3.0.0 Rasa SDK Version : 3.2.0 Python Version : 3.8.7 Operating System : Windows-10-10.0.22000-SP0

config.yml

language: en

pipeline:

• name: WhitespaceTokenizer
• name: LanguageModelFeaturizer model_name: bert model_weights: rasa/LaBSE cache_dir: null
• name: RegexFeaturizer
• name: LexicalSyntacticFeaturizer
• name: CountVectorsFeaturizer
• name: CountVectorsFeaturizer analyzer: char_wb min_ngram: 1 max_ngram: 4
• name: DIETClassifier epochs: 100 constrain_similarities: true
• name: EntitySynonymMapper
• name: ResponseSelector epochs: 100 constrain_similarities: true
• name: FallbackClassifier threshold: 0.3 ambiguity_threshold: 0.1

policies:

Command / Request

No response

Relevant log output

Traceback (most recent call last):
  File "c:\users\navid\documents\development\try-rasa-chatbot\venv\lib\site-packages\rasa\engine\graph.py", line 390, in _load_component
    self._component: GraphComponent = constructor(  # type: ignore[no-redef]
  File "c:\users\navid\documents\development\try-rasa-chatbot\venv\lib\site-packages\rasa\engine\graph.py", line 217, in load
    return cls.create(config, model_storage, resource, execution_context)
  File "c:\users\navid\documents\development\try-rasa-chatbot\venv\lib\site-packages\rasa\nlu\featurizers\dense_featurizer\lm_featurizer.py", line 98, in create
    return cls(config, execution_context)
  File "c:\users\navid\documents\development\try-rasa-chatbot\venv\lib\site-packages\rasa\nlu\featurizers\dense_featurizer\lm_featurizer.py", line 65, in __init__
    self._load_model_instance()
  File "c:\users\navid\documents\development\try-rasa-chatbot\venv\lib\site-packages\rasa\nlu\featurizers\dense_featurizer\lm_featurizer.py", line 153, in _load_model_instance  
    self.model = model_class_dict[self.model_name].from_pretrained(  # type: ignore[no-untyped-call] # noqa: E501
  File "c:\users\navid\documents\development\try-rasa-chatbot\venv\lib\site-packages\transformers\modeling_tf_utils.py", line 1546, in from_pretrained
    model(model.dummy_inputs)  # build the network with dummy inputs
  File "c:\users\navid\documents\development\try-rasa-chatbot\venv\lib\site-packages\keras\utils\traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "c:\users\navid\documents\development\try-rasa-chatbot\venv\lib\site-packages\transformers\models\bert\modeling_tf_bert.py", line 1129, in call
    outputs = self.bert(
  File "c:\users\navid\documents\development\try-rasa-chatbot\venv\lib\site-packages\transformers\models\bert\modeling_tf_bert.py", line 789, in call
    embedding_output = self.embeddings(
  File "c:\users\navid\documents\development\try-rasa-chatbot\venv\lib\site-packages\transformers\models\bert\modeling_tf_bert.py", line 150, in build
    self.weight = self.add_weight(
tensorflow.python.framework.errors_impl.ResourceExhaustedError: Exception encountered when calling layer "bert" (type TFBertMainLayer).

failed to allocate memory [Op:AddV2]

Call arguments received:
  • input_ids=tf.Tensor(shape=(3, 5), dtype=int32)
  • attention_mask=None
  • token_type_ids=None
  • position_ids=None
  • head_mask=None
  • inputs_embeds=None
  • encoder_hidden_states=None
  • encoder_attention_mask=None
  • past_key_values=None
  • use_cache=True
  • output_attentions=False
  • output_hidden_states=False
  • return_dict=True
  • training=False
  • kwargs=<class 'inspect._empty'>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\navid\.pyenv\pyenv-win\versions\3.8.7\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\navid\.pyenv\pyenv-win\versions\3.8.7\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\navid\Documents\Development\try-rasa-chatbot\venv\Scripts\rasa.exe\__main__.py", line 7, in <module>
  File "c:\users\navid\documents\development\try-rasa-chatbot\venv\lib\site-packages\rasa\__main__.py", line 110, in main
    cmdline_arguments.func(cmdline_arguments)
  File "c:\users\navid\documents\development\try-rasa-chatbot\venv\lib\site-packages\rasa\cli\train.py", line 62, in <lambda>
    train_parser.set_defaults(func=lambda args: run_training(args, can_exit=True))
  File "c:\users\navid\documents\development\try-rasa-chatbot\venv\lib\site-packages\rasa\cli\train.py", line 94, in run_training
    training_result = train_all(
  File "c:\users\navid\documents\development\try-rasa-chatbot\venv\lib\site-packages\rasa\api.py", line 105, in train
    return train(
  File "c:\users\navid\documents\development\try-rasa-chatbot\venv\lib\site-packages\rasa\model_training.py", line 160, in train
    return _train_graph(
  File "c:\users\navid\documents\development\try-rasa-chatbot\venv\lib\site-packages\rasa\model_training.py", line 234, in _train_graph
    trainer.train(
  File "c:\users\navid\documents\development\try-rasa-chatbot\venv\lib\site-packages\rasa\engine\training\graph_trainer.py", line 105, in train
    graph_runner.run(inputs={PLACEHOLDER_IMPORTER: importer})
  File "c:\users\navid\documents\development\try-rasa-chatbot\venv\lib\site-packages\rasa\engine\runner\dask.py", line 101, in run
    dask_result = dask.get(run_graph, run_targets)
  File "c:\users\navid\documents\development\try-rasa-chatbot\venv\lib\site-packages\dask\local.py", line 552, in get_sync
    return get_async(
  File "c:\users\navid\documents\development\try-rasa-chatbot\venv\lib\site-packages\dask\local.py", line 495, in get_async
    for key, res_info, failed in queue_get(queue).result():
  File "C:\Users\navid\.pyenv\pyenv-win\versions\3.8.7\lib\concurrent\futures\_base.py", line 432, in result
    return self.__get_result()
  File "C:\Users\navid\.pyenv\pyenv-win\versions\3.8.7\lib\concurrent\futures\_base.py", line 388, in __get_result
    raise self._exception
  File "c:\users\navid\documents\development\try-rasa-chatbot\venv\lib\site-packages\dask\local.py", line 537, in submit
    fut.set_result(fn(*args, **kwargs))
  File "c:\users\navid\documents\development\try-rasa-chatbot\venv\lib\site-packages\dask\local.py", line 233, in batch_execute_tasks
    return [execute_task(*a) for a in it]
  File "c:\users\navid\documents\development\try-rasa-chatbot\venv\lib\site-packages\dask\local.py", line 233, in <listcomp>
    return [execute_task(*a) for a in it]
  File "c:\users\navid\documents\development\try-rasa-chatbot\venv\lib\site-packages\dask\local.py", line 224, in execute_task
    result = pack_exception(e, dumps)
  File "c:\users\navid\documents\development\try-rasa-chatbot\venv\lib\site-packages\dask\local.py", line 219, in execute_task
    result = _execute_task(task, data)
  File "c:\users\navid\documents\development\try-rasa-chatbot\venv\lib\site-packages\dask\core.py", line 119, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "c:\users\navid\documents\development\try-rasa-chatbot\venv\lib\site-packages\rasa\engine\graph.py", line 451, in __call__
    self._load_component(**constructor_kwargs)
  File "c:\users\navid\documents\development\try-rasa-chatbot\venv\lib\site-packages\rasa\engine\graph.py", line 403, in _load_component
    raise GraphComponentException(
rasa.engine.exceptions.GraphComponentException: Error initializing graph component for node run_LanguageModelFeaturizer1.

[ancalita]

@navidved Seems that you're running into an out-of-memory issue and you might need to run the program on a different machine with more memory.