CrossValidation: UnboundLocalError: local variable 'intent_evaluation' referenced before assignment

[Mazyod]

Rasa Open Source version

3.1.0

Rasa SDK version

No response

Rasa X version

No response

Python version

3.9

What operating system are you using?

Windows

What happened?

We call Rasa functions directly. Calling await run_nlu_test_async(...) lead to the following issue:

UnboundLocalError: local variable 'intent_evaluation' referenced before assignment

Looking at the code, the minimum suggested improvement is handling the cases where the if statement is not executed, and either passing an empty dictionary or raising a useful error to the user:

https://github.com/RasaHQ/rasa/blob/a78e319d3a3adb46fd6a75275e76645df0d728fc/rasa/nlu/test.py#L1645-L1654

https://github.com/RasaHQ/rasa/blob/a78e319d3a3adb46fd6a75275e76645df0d728fc/rasa/nlu/test.py#L1682-L1685

Command / Request

from rasa.cli.test import run_nlu_test_async

# attempt to merge with test model function proved more trouble than it's worth
await run_nlu_test_async(
    config=str(workspace.input.config_file),
    data_path=str(workspace.input.nlu_file),
    models_path="",  # :(
    output_dir=str(workspace.output.results_dir),
    cross_validation=True,
    percentages=[],  # :(
    runs=0,  # :(
    no_errors=not config["errors"],
    domain_path=None,  # :(
    all_args={
        "folds": config["folds"],
        "disable_plotting": not config["plots"],
        "successes": config["successes"],
    },
)

Relevant log output

Traceback (most recent call last):
  File "...py", line 171, in run
    await worker_handle
ray.exceptions.RayTaskError(UnboundLocalError): ray::worker_process() (pid=26928, ip=127.0.0.1)
  File "python\ray\_raylet.pyx", line 640, in ray._raylet.execute_task
  File "python\ray\_raylet.pyx", line 644, in ray._raylet.execute_task
  File "...py", line 76, in worker_process
    return asyncio.get_event_loop().run_until_complete(result)
  File "...py", line 647, in run_until_complete
    return future.result()
  File "...py", line 298, in _cross_validate_model
    await run_nlu_test_async(
  File "...\venv\lib\site-packages\rasa\cli\test.py", line 232, in run_nlu_test_async
    await perform_nlu_cross_validation(config_dict, nlu_data, output, all_args)
  File "...\venv\lib\site-packages\rasa\model_testing.py", line 322, in perform_nlu_cross_validation
    results, entity_results, response_selection_results = await cross_validate(
  File "...\venv\lib\site-packages\rasa\nlu\test.py", line 1684, in cross_validate
    dict(intent_train_metrics), dict(intent_test_metrics), intent_evaluation
UnboundLocalError: local variable 'intent_evaluation' referenced before assignment

[Mazyod]

It seems clear now we should pass the domain file, but the error is misleading.

[sara-tagger]

Thanks for the issue, @amn41 will get back to you about it soon!

You may find help in the docs and the forum, too 🤗

[amn41]

thanks - just to clarify, when you run run_nlu_test_async without providing a domain, the function tries to execute and runs into an error, correct? I'll change this to an enhancement request then, to either make this a required parameter or handle the absence of a domain file gracefully.

[Mazyod]

@amn41 We use the same function to cross validate an NLU only model, and calling the function as is above works fine. Recently, we started experimenting with e2e pipelines, and saw this issue with an e2e trained model.

[amn41]

would love to hear how your e2e results are going! my email is on my github profile - would love to chat.

[ronaldgevern]

An UnboundLocalError is raised when a local variable is referenced before it has been assigned. In most cases this will occur when trying to modify a local variable before it is actually assigned within the local scope. Python doesn't have variable declarations, so it has to figure out the scope of variables itself. It does so by a simple rule: If there is an assignment to a variable inside a function, that variable is considered local.

Python has lexical scoping by default, which means that although an enclosed scope can access values in its enclosing scope, it cannot modify them (unless they're declared global with the global keyword). A closure binds values in the enclosing environment to names in the local environment. The local environment can then use the bound value, and even reassign that name to something else, but it can't modify the binding in the enclosing environment. UnboundLocalError happend because when python sees an assignment inside a function then it considers that variable as local variable and will not fetch its value from enclosing or global scope when we execute the function. However, to modify a global variable inside a function, you must use the global keyword.

[sync-by-unito[bot]]

➤ Maxime Verger commented:

:bulb: Heads up! We're moving issues to Jira: https://rasa-open-source.atlassian.net/browse/OSS.

From now on, this Jira board is the place where you can browse (without an account) and create issues (you'll need a free Jira account for that). This GitHub issue has already been migrated to Jira and will be closed on January 9th, 2023. Do not forget to subscribe to the corresponding Jira issue!

:arrow_right: More information in the forum: https://forum.rasa.com/t/migration-of-rasa-oss-issues-to-jira/56569.