Reproduce Comparison Question Results

[xanhho]

Hi, thank you for your repository.

In my understanding, to obtain the scores for the comparison questions, we need to do the following steps:

1. Prepare sub-questions data

Download your code and data here, then run it, then prepare two files for the single-hop models (dev_c.1.json and dev_c.2.json).

2. Single-hop RC Model

Run the single-hop models for these files. For example:

python3 main.py --do_predict --output_dir out/compare_1 \
--predict_file data/decomposed/dev_c.1.json \
--init_checkpoint model/onehop/model1.pt,model/onehop/model2.pt,model/onehop/model3.pt \
--n_best_size 4

3. Obtain the final answer for each sample

From your code, I think we have 8 operators. I can use some heuristics to obtain the final results. If it is ok, could you share your file on obtaining the final score here for reproducing the results?

Also, please correct me if my understanding about your model is not correct.

Thank you for your support.