`evaluate.load` not loading all files in the repo

[loubnabnl]

evaluate.load only loads the files that are imported in the main file, so a user will need to include the import in the main file even if they don't use it there for the files to be loaded, it might be nice to include this in the docs.

An example here: https://huggingface.co/spaces/loubnabnl/apps_metric/blob/main/utils.py#L5 and the error I got before I included the import in the main file:

from .utils import compute_metrics
  File "/Users/loubnabenallal/.cache/huggingface/modules/evaluate_modules/metrics/loubnabnl--apps_metric/24ed11d3adfebede98beb04ac2c729404f305f66064c2b940c893f510452cea9/utils.py", line 5, in <module>
    from .testing_util import run_test
ModuleNotFoundError: No module named 'evaluate_modules.metrics.loubnabnl--apps_metric.24ed11d3adfebede98beb04ac2c729404f305f66064c2b940c893f510452cea9.testing_util'

And after checking the files that were loaded, testing_util wasn't there:

ls /Users/loubnabenallal/.cache/huggingface/modules/evaluate_modules/metrics/loubnabnl--apps_metric/24ed11d3adfebede98beb04ac2c729404f305f66064c2b940c893f510452cea9/

__init__.py             __pycache__             apps_metric.json        apps_metric.py          utils.py

• evaluate version 0.1.2
• datasets version 2.1.0

[mathemakitten]

Seconding that this seems like it'd be useful! I recently tried to load a custom metric which opened up a JSON file for configuration, but the JSON file wasn't loaded. I think this might not be a "canonical" use case though, so perhaps optional to actually support.

[lvwerra]

Note that you can use the _download_and_prepare method to download any additional files:

https://github.com/huggingface/evaluate/blob/eaf34a7d04e7ab3e6155a046f6d7fda01d9ead84/templates/%7B%7B%20cookiecutter.module_slug%20%7D%7D/%7B%7B%20cookiecutter.module_slug%20%7D%7D.py#L84