Open-Assistant icon indicating copy to clipboard operation
Open-Assistant copied to clipboard

In "Label Assistant Reply" subdivide "Quality" into "Accuracy" and f.e. "Grammar"

Open ChaoticHuman opened this issue 2 years ago • 2 comments

I think it would be a good idea to subdivide the "Quality"-category in "Label Assistant Reply" into "Accuracy" and f.e. "Grammar" if that is possible because the term "Quality" is a relatively vague term. And I also think it would generally be a good idea to focus the training of Open Assistant more on accuracy as inaccurate outputs are one of the biggest issues of competitors like ChatGPT, etc.

Ok, I must admit though it's not always easy for people to asses the accuracy of an output and it's admittedly also possible that people might bring in their own biases meaning I would still keep answering the new categories "Accuracy" and "Grammar" optional and would perhaps suggest to the main-developers of Open Assistant to see the user-contributed "Accuracy"-assesment as merely a first step in checking the accuracy of assistant replies.

ChaoticHuman avatar Feb 05 '23 05:02 ChaoticHuman

Ok, I was thinking about closing that issue again because I just found out that the dashboard also includes a function sometimes popping up called something like "Rank assistant reply" where you are apparently supposed to rank assistant replies based on accuracy however as the issue got two likes I will keep it open for now.

ChaoticHuman avatar Feb 05 '23 07:02 ChaoticHuman

I don't think it's practical to change the labelling options with so much data already submitted, but maybe others have other views so we can leave this open for any further discussion

olliestanley avatar Feb 06 '23 21:02 olliestanley

Going to close this as covered by #1412

olliestanley avatar Jun 12 '23 17:06 olliestanley