autolabel icon indicating copy to clipboard operation
autolabel copied to clipboard

[Feature Request]: Check LLM label against multiple-choice options for question answering tasks

Open Tyrest opened this issue 2 years ago • 0 comments
trafficstars

Is your feature request related to a problem? Please describe. Sometimes, during a question_answering task, the LLM will output something that is not part of the multiple-choice options for the question (if they are given like in sciq). We want to check every LLM label to make sure that it is one of the provided options and if not, we should mark the row's succesfully_labeled as False.

could we add this for question answering as well where there are a set of answers for each question?

Originally posted by @rajasbansal in https://github.com/refuel-ai/autolabel/pull/345#pullrequestreview-1489194691

Describe the solution you'd like If multiple-choice options are given, check each LLM label during question_answering tasks to make sure it is one of the provided options and mark succesfully_labeled as False if it is not.

Additional context #345 implements something similar for classification and entity_matching tasks.

Tyrest avatar Jun 20 '23 23:06 Tyrest