LAION-AI
Labeling: Standards & Guidelines
Currently, not only the evaluations, but also the definitions of the terms are often very dependent on a subjective perception. It would be advantageous if there were universally valid definitions of these terms.
Ideally, this would be combined with some examples so that a general understanding of the criteria can be established. If this would be embedded on the Training GUI or at least interlinked, that would be super helpful for everyone and potentially increases the rating quality.
A starting point could be: https://github.com/LAION-AI/Open-Assistant/blob/main/docs/docs/guides/data_collection.md#ranking-and-classifying-prompts
A starting point could be: https://github.com/LAION-AI/Open-Assistant/blob/main/docs/docs/guides/data_collection.md#ranking-and-classifying-prompts
Yes - if anyone wants to add docs with further explanation/detail please add it to the file linked here.
Also I think that it should be a quick tutorial and tests like https://eyewire.org/ have (many of users don't really get an idea of what this thing should do, how restrictions would work and other stuff, initial tutorial and test should make it more clear)