Issue in `call_predict`: error while getting predictions from trained models

[swag2198]

Describe the bug We are trying to reproduce the experiments for the paper Comparing Test Sets with Item Response Theory, and we were following the workflow mentioned in this README for the preprocessing of datasets, training models, generating responses, etc. We have successfully trained most of the models at our end, but now facing an issue while getting predictions from those models.

To Reproduce

1. jiant version: 2.1.4
2. Environment: A100 40GB

We ran the call_predict script. We did not find any sb_predict_results.sbatch file, and changed L21 to call the run_predict_results.sbatch file instead. For example, the exact command we ran for copa and for bert-base-cased model is:

python jiant/irt_scripts/call_predict.py bert-base-cased copa $(pwd)

This produced the following error:

pre args =  Namespace(ZZoverrides=['model_path'], ZZsrc=None)

Traceback (most recent call last):

  File "/dccstor/irlirteval/irt/jiant/jiant/proj/main/runscript.py", line 256, in <module>

    main()

  File "/dccstor/irlirteval/irt/jiant/jiant/proj/main/runscript.py", line 246, in main

    run_loop(RunConfiguration.default_run_cli(cl_args=cl_args))

  File "/dccstor/irlirteval/irt/jiant/jiant/utils/zconf/core.py", line 237, in default_run_cli

    return cls.run_cli_json_prepend(cl_args=cl_args, prog=prog, description=description)

  File "/dccstor/irlirteval/irt/jiant/jiant/utils/zconf/core.py", line 174, in run_cli_json_prepend

    result = cls.run_from_parser_json_prepend(parser=parser, cl_args=cl_args)

  File "/dccstor/irlirteval/irt/jiant/jiant/utils/zconf/core.py", line 217, in run_from_parser_json_prepend

    assert pre_args.ZZoverrides is None

AssertionError

An attempt to fix We tried to fix the zzoverrides error by modifying the run_predict_results.sbatch as follows:

python $JIANT_PATH/proj/main/runscript.py run   \
    --hf_pretrained_model_name_or_path ${model} \
    --model_path ${MODELS_DIR}/${SHORT_MODEL_NAME}/model/model.p \
    --model_config_path ${MODELS_DIR}/${SHORT_MODEL_NAME}/model/config.json \
    --jiant_task_container_config_path ${BASE_PATH}/experiments/run_config_dir/taskmaster/${model}/${task}_${config_no}/${task}.json  \
    --model_load_mode all --model_path ${BASE_PATH}/experiments/output_dir/taskmaster_${model}/${task}/config_${config_no}/${model_path}  \
    --output_dir  ${BASE_PATH}/experiments/predict_files/${model}/${task}_config_${config_no}_${model_path} --ZZoverrides model_path model_config_path hf_pretrained_model_name_or_path --write_val_preds \
    --ZZsrc ${MODELS_DIR}/${SHORT_MODEL_NAME}/config.json --do_val --do_save --force_overwrite

instead of this code block. This time it produced empty predictions.

Expected behavior Prediction files (.p) in experiments/predict_files/ folder, so that we could run the post-processing command python jiant/irt_scripts/postprocess_predictions.py $(pwd).