LAION-AI
Limit the number of responses in ranking task
Currently ranking tasks are pretty difficult when there's too many nodes at the level. Here's an extreme example posted on Discord:
Personally I find it difficult to order even 3 multiple-paragraph responses.
Suggestion: have multiple stages - thumbs up/down the answers, remove thumbs down, if there's more than 3 left then pick the best 3, then order the remaining ones.
To clarify this, I think we already try to limit the number of replies to 3 or 4 but if users have the task concurrently it can go over. So what we need is a solution which prevents this in future
100ecbe2d53c35c00279b2d1d9aa31754ec265e0 should address this to some extent I believe. If anyone still has problems with this please comment here
We should no longer be seeing huge ranking tasks and I haven't seen any reports of unreasonable amounts recently so this issue seems to be resolved. Feel free to comment again if you are still seeing it.