RankLib icon indicating copy to clipboard operation
RankLib copied to clipboard

Published 20 hours ago • verified publisher icon jattenberg

Issue with metric

Open shashankg7 opened this issue 8 years ago • 2 comments

Hi,

Thanks for hosting this library of github. I am trying it on my custom dataset. It works fine for ranking, but when I pass relevance scores > 1 (like 3,2,1 levels), it gives MAP, NDCG as 1 always.

I am not sure what is wrong. Can you please help me out?

Thanks

shashankg7 avatar Dec 21 '16 17:12 shashankg7

Hi @shashankg7 I'm not actively maintaining this. Seems like you need a mapping from scores to rankings. I'm happy to take a pull request if you could build one, it should be straight forward, basically just sorting.

jattenberg avatar Dec 23 '16 15:12 jattenberg

HI @jattenberg . That problem is resolved. I have a small doubt related to RankLib. It would be really useful if you could resolve my doubt.

My problem involved working with set of queries and set of relevant documents corresponding to each query i.e. I have dataset with only relevance level of 1 and no zeros.

For training I can randomly sample o(irrelevant) samples from my corpus, but for testing I will only have (q,d) pairs with +1 label.

Does it make sense to evaluate on such dataset?

shashankg7 avatar Dec 24 '16 14:12 shashankg7