How can I test the model 7_STAR_ORA-P-D-last-step-raw

[HamLaertes]

Hi, Thanks for your codes and datasets, I have questions about how to test models (such as 7_STAR_ORA-P-D-last-step-raw) that depend on gold paragraphs and decompositions. Since the two dependencies are missed in the test file, directly running python run_scripts/predict.py --model ./7_STAR_ORA-P-D-last-step-raw.tar.gz --data test.json -g 0 raises errors:

    question = decomposition[-1]["question"]
TypeError: 'NoneType' object is not subscriptable

What should I do to test those models? Thanks!!

[HamLaertes]

Hi, @eladsegal I have some thoughts on my questions. Those models are called 'ORACLE' models. 'ORACLE' may indicate they use annotated test data, but only to discuss some properties of StrategyQA, not to compare with other models. Am I correct? Looking forwards to your replies!

I have another question about retrieving the paragraphs using questions/decompositions. The paper says the retrieving is done based on the BM25 scores. But I cannot find whether the score is calculated in the codes.

Although I am not familiar with Elasticsearche and you've provided the retrieved results, as the paper points out, retrieving plays an important step, I feel it is necessary for me to thoroughly understand how the retrieving is done in your codes. Looking forwards to your replies!

Thanks!