Using JIANT to test model bias on winogender task

[mzakizadeh]

Describe the bug I attempted to test the performance of bert-large-uncased on the Winogender task using JIANT. For this purpose, I utilized a straightforward code snippet to conduct the experiment. However, during execution, I encountered an assertion error. My hunch is that this error may have resulted from an update that potentially disrupted the compatibility with the dataset.

To Reproduce

1. I used the latest version of the library available on pip (which I believe is version 2.2.0)
2. Execute the following code:

import os
import jiant.utils.python.io as py_io
import jiant.proj.simple.runscript as run
import jiant.scripts.download_data.runscript as downloader

TASK_NAME = "superglue_winogender_diagnostics"

HF_PRETRAINED_MODEL_NAME = "bert-large-uncased"

MODEL_NAME = HF_PRETRAINED_MODEL_NAME.split("/")[-1]
RUN_NAME = f"simple_{TASK_NAME}_{MODEL_NAME}"
EXP_DIR = "exp"
DATA_DIR = "exp/tasks"

os.makedirs(DATA_DIR, exist_ok=True)
os.makedirs(EXP_DIR, exist_ok=True)

downloader.download_data([TASK_NAME], DATA_DIR)

args = run.RunConfiguration(
    run_name=RUN_NAME,
    exp_dir=EXP_DIR,
    data_dir=DATA_DIR,
    hf_pretrained_model_name_or_path=HF_PRETRAINED_MODEL_NAME,
    tasks=TASK_NAME,
    train_batch_size=16,
    num_train_epochs=1
)
run.run_simple(args)

Expected behavior The code should run without any assertion errors and evaluate the bert-large-uncased model on the Winogender task.

Actual Behavior: The code throws an assertion error, preventing successful evaluation.

Traceback (most recent call last):
  File "test.py", line 40, in <module>
    args = simple_run.RunConfiguration.from_json_path(os.path.join(EXP_DIR, "runs", RUN_NAME, "simple_run_config.json"))
  File "/home/zakizadeh/.conda/envs/effectiveness/lib/python3.7/site-packages/jiant/utils/zconf/core.py", line 121, in from_json_path
    return cls.from_json(f.read())
  File "/home/zakizadeh/.conda/envs/effectiveness/lib/python3.7/site-packages/jiant/utils/zconf/core.py", line 116, in from_json
    return cls(**json.loads(json_string))
  File "<attrs generated init jiant.proj.simple.runscript.RunConfiguration>", line 39, in __init__
  File "/home/zakizadeh/.conda/envs/effectiveness/lib/python3.7/site-packages/jiant/utils/zconf/core.py", line 284, in __attrs_post_init__
    self._post_init()
  File "/home/zakizadeh/.conda/envs/effectiveness/lib/python3.7/site-packages/jiant/proj/simple/runscript.py", line 85, in _post_init
    ), "Tasks must be same as train_tasks/val_tasks/test_tasks if both are present"
AssertionError: Tasks must be same as train_tasks/val_tasks/test_tasks if both are present