argilla icon indicating copy to clipboard operation
argilla copied to clipboard

[PROPOSAL] Allow an arbitrary number of rankings for `RankingQuestion`

Open alvarobartt opened this issue 11 months ago • 2 comments

Describe your proposal

As of now, the RankingQuestion doesn't let you define the number of ranks you want to use, as that just matches the amount of options you define e.g. you define 5 options then the ranking will go from 1 to 5, but maybe we want to have just 1-2 and 5 options, and that's currently not possible.

So on, this proposal states that we should be able to at least define the number of ranks we want to use instead of just matching the options. This could be useful for scenarios where we want to group the options in N categories (not explicitly being a ranking), when we want to put in the first rank the best options and in the last rank the worst ones, etc.

Ideally this would use the same number of options for the ranks by default, but also allowing the user to initially define the number of ranks as an integer e.g. ranks=3 implies that [1, 2, 3].

For the moment we should initially support that, but maybe eventually in the future also to specify the ranks as strings instead of sequential integers, but that may even be a different question towards the user.

This could also be beneficial for data annotation towards DPO or RLHF fine-tuning of existing LLMs cc @tomaarsen

alvarobartt avatar Aug 02 '23 08:08 alvarobartt