Meaning of QL output

[zhenduow]

Hi,

Thank you for putting together this amazing instruction. It is easy to follow and works smooth. I have no problem getting the output in run_QL.ipynb. However, I find them hard to understand:

1. I get the middle column is the ranking score of some sort.
2. The first column seems like the clueweb09 index and the only information source for computing ranking performance, but how do I get the relevance labels of them for the query 1-1-1?

Can you please explain the output dataframe, especially what the columns mean and how to compute the ranking metrics?

Thank you very much!

[aliannejadi]

Hi, Thank you for your questions and interest.

1. yes that is the ranking.
2. The qrel can be made easily from the qrels published by TREC Web track. For each query, you have the topic and facet ID and can use that to get the relevance judgements of TREC Web track.

With the qrels, you can easily compute the metrics using trec_eval or pytrec_eval.

Hope this helps, Mohammad