learn to rank with Bert (point-wise) ranking

[MiladAlshomary]

Hello all,

I have been experimenting with the model "learn to rank with Bert". The list-wise approach works well. However, I wanted to evaluate the performance of the model in the pair-wise and point-wise setting.

To do so, I have tried to use the tfr_bert_example by modifying listwise_inference=True into False and using sigmoid_cross_entropy_loss as a loss function, but I got an error regarding the dimensions of the input being wrong.

My question is: Does the implementation of tfr Bart here supports point-wise or pair-wise setting? if not, could you point out how that can be modified to do so?

Thanks a lot for your help!

[HongleiZhuang]

Hi @MiladAlshomary. You can directly use the sigmoid cross-entropy loss without modifying listwise_inference. As the comment suggested, only listwise inference is currently supported for keras ranking network.

[MiladAlshomary]

Hi @HongleiZhuang, thanks for your reply!

More important for me is to evaluate the point-wise and pair-wise ltr approaches rather than loss_funtions. Only changing the loss function to cross-entropy would not make it point-wise right? To do so then I need to implement my own ranking network?

Thanks!

[HongleiZhuang]

Hi @MiladAlshomary. Using cross-entropy loss essentially makes the model to be trained with a point-wise loss, i.e., during training, the loss of an example is only related to the example per se but not any other examples in the same list. So if you are comparing model performance based on whether the model is trained on point-wise loss vs. pair-wise loss, changing the loss would be sufficient.

For inference, the current TFR-BERT model would be a point-wise scorer anyway regardless of the loss function you use to train it, i.e., the inference of the ranking score is not related to any other examples in the same list.

With listwise_inference=True, it means you essentially need to feed an ELWC (even if it is a size-1 ELWC) into the savedmodel to obtain ranking scores. You cannot just feed a tf.Example like you do with other tf-ranking models with listwise_inference=False.

[MiladAlshomary]

Thanks a lot, for the clarification @HongleiZhuang ,

So, if I want to train the model in a pair-wise setting, then I have to use one of the pairwise_* loss functions?

[HongleiZhuang]

Hi @MiladAlshomary,

Yes. Using one of the pairwise_*loss functions would allow the model to be trained in a pair-wise setting. It means the model would look at other examples in the same list during training (but not during inference).

[MiladAlshomary]

Hi @HongleiZhuang ,

I see that, as you explained, that upon inference, each document gets its score independent from the other. Any reason why computing the scores of sentences in inference time can't be configured to use listwise/pairwise setting?

[HongleiZhuang]

Hi @MiladAlshomary ,

Sorry for the late reply. Correct --- each document gets its score independent from the other during inference. It is definitely a very good question on whether we can compute scores with listwise/pairwise setting, and it is something we will look into.

Thank you!