[Task] Compute ranking metrics for models trained with a binary classification task

[sararb]

Description

This goal is to be able to compute ranking metrics for models trained with the binary task.

Additional context One possible solution is to have a post-processing step that groups the scores by USER_ID (or another column like session-id), sorts them, and feeds this list of scores together with the related list of binary targets to the ranking metric.

[marcromeyn]

That’s how we did it at Spotify as well, we presented that system here. It would be very straightforward to replicate that here.