autolabel
autolabel copied to clipboard
[Feature Request]: Check LLM label against multiple-choice options for question answering tasks
Is your feature request related to a problem? Please describe.
Sometimes, during a question_answering task, the LLM will output something that is not part of the multiple-choice options for the question (if they are given like in sciq). We want to check every LLM label to make sure that it is one of the provided options and if not, we should mark the row's succesfully_labeled as False.
could we add this for question answering as well where there are a set of answers for each question?
Originally posted by @rajasbansal in https://github.com/refuel-ai/autolabel/pull/345#pullrequestreview-1489194691
Describe the solution you'd like
If multiple-choice options are given, check each LLM label during question_answering tasks to make sure it is one of the provided options and mark succesfully_labeled as False if it is not.
Additional context
#345 implements something similar for classification and entity_matching tasks.