Open-Assistant
Open-Assistant copied to clipboard
Limit the number of responses in ranking task
Currently ranking tasks are pretty difficult when there's too many nodes at the level. Here's an extreme example posted on Discord:
Personally I find it difficult to order even 3 multiple-paragraph responses.
Suggestion: have multiple stages - thumbs up/down the answers, remove thumbs down, if there's more than 3 left then pick the best 3, then order the remaining ones.
To clarify this, I think we already try to limit the number of replies to 3 or 4 but if users have the task concurrently it can go over. So what we need is a solution which prevents this in future
100ecbe2d53c35c00279b2d1d9aa31754ec265e0 should address this to some extent I believe. If anyone still has problems with this please comment here
We should no longer be seeing huge ranking tasks and I haven't seen any reports of unreasonable amounts recently so this issue seems to be resolved. Feel free to comment again if you are still seeing it.