question on lambdarank_num_pair_per_sample

[randomgambit]

Hello

Thanks for this amazing package. I have a question regarding the parameters I see for Xgbranker. I have been reading the documentation many times (https://xgboost.readthedocs.io/en/latest/tutorials/learning_to_rank.html ) and I am not sure if I am setting the lambdarank_num_pair_per_sample correctly.

My setup is simple, about 300 groups, eqch group has 50 documents, and about half of them (in each group) have relevance = 1, the rest is = 0 (so I use rank::map as metric). When I train the ranker, shouldnt we use all possible pairs in each group (given my small sample)? That is, setting lambdarank_num_pair_per_sample to a very high value? This seems at odds with the documentation. I am using the mean, not topk setting.

Thanks for clarifying this point.

[trivialfis]

Hi, the document provides guidelines for selecting the initial setup and what matters in a dataset (effective pairs). You will have to run experiments to confirm. The ranking objectives have the most hyperparameters of all objectives. We don't want to have this many tunable variables, but we couldn't remove them because it's just difficult to know in advance.

[randomgambit]

thank you so much for the quick reply! what I meant is whether it makes sense to have a very large number of comparisons per query. Is this reasonable at all given the algorithm?

Also, for my setting in broad terms, do you think the rank::pairwise makes sense? Again I am trying to understand the guidelines in the doc. Any insights super appreciated!! thanks!

[trivialfis]

whether it makes sense to have a very large number of comparisons per query.

Yes, it makes sense to try IF your model doesn't converge.

Also, for my setting in broad terms, do you think the rank::pairwise makes sense

I wouldn't put it very high on the to-do list for things to try. If I were to use a multi-level objective, I would pick NDCG first. Then try MAP if my target metric is MAP. The lambdamart scaling factor $\Delta Z$ usually works (despite the lack of proof).

[trivialfis]

Since your dataset is relatively small, using the mean setting is a very good start. From experience, the sampling pairs don't have to be very large for binary labels. Your dataset has balanced labels, which should generate a sufficient number of pairs. I would start small with 1 for experimental purposes.

[randomgambit]

ha super interesting, thank you! however, this is opposite of what I would have imagined: setting num_pair to 1 would only sample ONE pair for each query? So that would generate very little data for the lambdas, no? Am I mistaken here? I am setting num_pairs to 100 at the moment...

[trivialfis]

Quoting the document:

XGBoost will randomly sample lambdarank_num_pair_per_sample pairs for each element in the query group .

Pair for each element.

[randomgambit]

Thank you so much @trivialfis , and sorry if this is obvious. So if we consider the example below:

query_ID    rank       variable
1              1         x1
1              1         x2
1              2         x3
1              3         x4
1              3         x5

could you please explain which pairs are used to compute the gradients with a given lambdarank_num_pair_per_sample? Say = 2. Again, sorry to bother you with this but I feel this is a critical point of the algorithm. To me, lambdarank will use every pair with different ranks, so that would be

(1, x1) vs (2, x3)
(1, x1) vs (3, x4)
(1, x1) vs (3, x5)
(1, x2) vs (2, x3)
(1, x2) vs (3, x4)
(1, x2) vs (3, x5)
(2, x3) vs (3, x4)
(2, x3) vs (3, x5)

is this what xgbranker is doing with the mean setting? how is the parameter above affecting this? Any clarification greatly appreciated @trivialfis !

Thanks

[trivialfis]

Since you have dug this deep, and my explanation skills are insufficient. You are not the first one to ask similar questions. Apologies for the confusion. Let's look into the code:

Since it's pairwise, we have two samples with the same query group for each pair. For random sampling (mean), the first sample is defined here:

https://github.com/dmlc/xgboost/blob/5546965639ca41dddfd69f15f6f1c68213e87767/src/objective/lambdarank_obj.h#L248

cnt is the number of samples in the group.

The second sample is defined here:

https://github.com/dmlc/xgboost/blob/5546965639ca41dddfd69f15f6f1c68213e87767/src/objective/lambdarank_obj.h#L266

n_samples is the lambdarank_num_pair_per_sample.

[trivialfis]

Please don't hesitate to ask if you have any more questions. Otherwise, it would be great if you could help improve the document from a user's perspective. Suggestions and PRs are welcome!