Open-Assistant icon indicating copy to clipboard operation
Open-Assistant copied to clipboard

Handle case when all prompts are factually inaccurate

Open JamesDConley opened this issue 2 years ago • 2 comments

Example image

In this case it would be good to allow the labeler to indicate all the answers are incorrect, or to add their own, factually correct answer.

JamesDConley avatar Jan 28 '23 21:01 JamesDConley

Best you currently can do is downgrade (thumbs down) and report the parent-message.

andreaskoepf avatar Jan 28 '23 21:01 andreaskoepf

For this you can optionally skip the task, downvote both answers. But also from a data training perspective, ranking them is still valuable.

I'm going to change this to a documentation bug and request that someone fills out these details in the relevant docs page.

To clear this I think it's sufficient to create a markdown file in docs/docs/tasks/rank_assistant_replies.md and add some instructions on what to do in corner cases. Then update website/src/components/Tasks/TaskTypes.tsx and fill out the appropriate help_link field.

fozziethebeat avatar Feb 01 '23 06:02 fozziethebeat

This has been implemented

olliestanley avatar Apr 29 '23 21:04 olliestanley