Open-Assistant
Open-Assistant copied to clipboard
LAION-AI
Scoring Adjustment
To help clear the prompt backlog, as per https://github.com/LAION-AI/Open-Assistant/issues/1659, this cuts the score weighting for creating a prompt in half. It also increases the score weighting for penning a good reply (it getting ranked first) by about 20%.
:x: pre-commit failed.
Please run pre-commit run --all-files locally and commit the changes.
Find more information in the repository's CONTRIBUTING.md
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
:x: pre-commit failed.
Please run pre-commit run --all-files locally and commit the changes.
Find more information in the repository's CONTRIBUTING.md
:x: pre-commit failed.
Please run pre-commit run --all-files locally and commit the changes.
Find more information in the repository's CONTRIBUTING.md
Just to confirm, self.rankings_total is the number of ranking tasks completed, right?
We cut the reward for initial prompts by about 90% a few days ago. I'm not sure we can resolve the problem in this way, I think the remaining issues are mostly just that prompting is what people want to do, and actually communicating to them that many prompts are not the most helpful action.
https://github.com/LAION-AI/Open-Assistant/commit/93d65bdbed73e154538453f8843274cbf416dce2
We cut the reward for initial prompts by about 90% a few days ago. I'm not sure we can resolve the problem in this way, I think the remaining issues are mostly just that prompting is what people want to do, and actually communicating to them that many prompts are not the most helpful action.
Yeah, I saw that too, but my thinking was that any little nudge in the right direction would help, and I didn't take into account behavioral/communication factors like those you mentioned.
If no one has anything more to add, I'll close the PR.
We can keep this open. It might be useful to reduce the score further - we can wait and see what Andreas thinks
I would suggest to add a new string env-variable with comma separated lang-codes of all languages for which the initial-prompt task should be disabled. A setting like "en,es" would then allow to deactivate the prompt for English and Spanish.
