scrolls pyrouge import error

[sshleifer]

when I try to run any _SCROLLSSummaryTask I get
(e.g. srolls_govreport)

ModuleNotFoundError: No module named 'datasets_modules.metrics.metricsscrolls_py.74e8c8051b3b18b1e72c49a033274c72e63ca708b628688199bb7001bb7a1884.rouge'

even after pip install -e. I'm in a lightly modified fork with datasets==2.0 installed

any idea?

Traceback

  File "/home/sam/character-tech/lm-evaluation-harness/lm_eval/utils.py", line 243, in _wrapper
    return fn(*args, **kwargs)
  File "/home/sam/character-tech/lm-evaluation-harness/lm_eval/evaluator.py", line 110, in simple_evaluate
    task_dict = lm_eval.tasks.get_task_dict(tasks)
  File "/home/sam/character-tech/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 425, in get_task_dict
    task_name_dict = {
  File "/home/sam/character-tech/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 426, in <dictcomp>                                                                             task_name: get_task(task_name)()
  File "/home/sam/character-tech/lm-evaluation-harness/lm_eval/tasks/scrolls.py", line 148, in __init__
    load_metric(_download_metric(), config_name=self.DATASET_NAME)
  File "/home/sam/.conda/envs/cai-env-kurt2/lib/python3.10/site-packages/datasets/load.py", line 1395, in load_metric
    metric_cls = import_main_class(metric_module, dataset=False)
  File "/home/sam/.conda/envs/cai-env-kurt2/lib/python3.10/site-packages/datasets/load.py", line 106, in import_main_class
    module = importlib.import_module(module_path)
  File "/home/sam/.conda/envs/cai-env-kurt2/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/sam/.cache/huggingface/modules/datasets_modules/metrics/metricsscrolls_py/74e8c8051b3b18b1e72c49a033274c72e63ca708b628688199bb7001bb7a1884/metricsscrolls_py.py", lin
e 8, in <module>
    from .rouge import compute_rouge, postprocess_text as rouge_postprocess_text  # From: https://huggingface.co/datasets/tau/scrolls/raw/main/metrics/rouge.py
ModuleNotFoundError: No module named 'datasets_modules.metrics.metricsscrolls_py.74e8c8051b3b18b1e72c49a033274c72e63ca708b628688199bb7001bb7a1884.rouge'

[StellaAthena]

Is this happening on main or big-refactor?

[sshleifer]

main

[sshleifer]

Is there a summary of what's changed on big-refactor somewhere?