biobert-pytorch icon indicating copy to clipboard operation
biobert-pytorch copied to clipboard

Error when running evaluation on BioASQ 4-6b datasets

Open abdallah197 opened this issue 4 years ago • 0 comments

@jhyuklee This happens only with datasets of 4b to 6b. the 7b dataset runs normally just like the example in the readme.the error message I think comes from converting the prediction file to a bioasq evaluation format

key error
qa['answers']

the command used to run the evaluation

python biobert-pytorch/question-answering/run_factoid.py   \
--model_type bert \
--train_file ${DATA_DIR}/BioASQ-train-factoid-${TEST}b.json \
--model_name_or_path deepset/bert-base-cased-squad2 \
--do_eval \
--predict_file ${DATA_DIR}/BioASQ-test-factoid-${TEST}b-${EXPER}.json  \
--golden_file ${DATA_DIR}/${TEST}B{EXPERR}_golden.json  \
--per_gpu_eval_batch_size ${BATCH_SIZE} \
--max_seq_length ${MAX_LENGTH} \
--seed ${SEED} \
--official_eval_dir ${OFFICIAL_DIR} --output_dir \
${SAVE_DIR} \
--overwrite_output_dir 

abdallah197 avatar Feb 03 '21 12:02 abdallah197