test.ref

[jingjojo]

Hello,

For the convolution neural tensor network, could you please explain what do the five values on each line inside test.ref file indicate?

Thanks.

[GauravBh1010tt]

Hi,

The test.ref file is the reference or the gold standard file which contains the true labels. This file is used to rank the performance of the model in terms of the MAP and MRR score. The 1st column is the question number, the 2nd column is the answer id, the 3rd is the score, 4th column is the rank and the last if the label. The 3rd and 4th column can be left as 0, according to the evaluation criteria. The most important column is the last one which ranks the answers according to the true labels. So basically, the ranking model would need to rank most important QA pairs in the top and least important at the bottom to get a high MAP and MRR score.

Hope it clears some of your doubt.

If you want to learn more about the evaluation then have a look at the TREC website.