haystack icon indicating copy to clipboard operation
haystack copied to clipboard

`Label` skips `answer` validation if `no_answer=None`

Open wochinge opened this issue 3 years ago • 0 comments

Describe the bug If the default value of no_answer is used the validation of no_answer here is skipped. Haystack then sets the default here which means that if the Label object is e.g. serialized and deserialized, the validation will kick in and potentially raise.

Error message Error that was thrown (if available)

Expected behavior The code should be like this so that the validation of no_answer, answer.context and answer.answer is happening correctly.

if self.answer is not None:
    if no_answer is None:
        # Set default before validation 
        no_answer = self.answer.answer == "" or self.answer.answer is None
        
    if no_answer == True:
        if self.answer.answer != "" or self.answer.context:
            raise ValueError(f"Got no_answer == True while there seems to be an possible Answer: {self.answer}")
    elif no_answer == False:
        if self.answer.answer == "":
            raise ValueError(
                f"Got no_answer == False while there seems to be no possible Answer: {self.answer}"
            )

Additional context Add any other context about the problem here, like document types / preprocessing steps / settings of reader etc.

To Reproduce

from haystack.schema import Label, Answer, Document

# doesn't raise
label = Label(answer=Answer(answer="", context="some value"), query="test", document=Document("dasd"), is_correct_answer=True, is_correct_document=True, origin="user-feedback")

# raises but shouldn't
Label.from_dict(label.to_dict())

FAQ Check

System:

  • OS:
  • GPU/CPU:
  • Haystack version (commit or version number):
  • DocumentStore:
  • Reader:
  • Retriever:

wochinge avatar Sep 07 '22 16:09 wochinge