OhadRubin
How to identify positive samples and negative samples?
In the paper, it mentions that the score is defined as follow. Is that calculated by summing the logprob of each token of the ground true y conditioned on (e,x)? If that were right, the score would not be between 0 and 1 as shown in the pic. In that case, what is the threshold used for identifying pos&neg samples? Appreciate it!
In the paper, it mentions that the score is defined as follow. Is that calculated by summing the logprob of each token of the ground true y conditioned on (e,x)?
Yes, you are correct.
If that were right, the score would not be between 0 and 1 as shown in the pic. I
Yes. I found it to be more intuitive to explain with probability instead of logprobs.... It means the same thing.
In that case, what is the threshold used for identifying pos&neg samples?
We just take the top-5 as positive and the bottom-5 as negative. Since all the candidates are from a BM25, the bottom-5 are good negatives
