edsl icon indicating copy to clipboard operation
edsl copied to clipboard

Cancel children of failed validation tasks

Open johnjosephhorton opened this issue 4 months ago • 0 comments

          A few notes: 
  1. If there a validation error, we don't add the answer to the current_answers dictionary. This causes piped answers to fail. I think we probably do not want to suppress these errors---if there is a validation error in the case of piping, we probably want to stop the whole process.
  2. We probably always want to bubble up validation errors so they are in the report, even if we don't stop execution because of them c.f., this line in Invigilators:
 except QuestionAnswerValidationError as e:
            answer = None
            comment = "The response was not valid."
            # if self.raise_validation_errors:
            exception_occurred = e

This is the problem - when a parent doesn't validate, the answer doesn't get passed so it can't run the question properly:

            try:
                response: EDSLResultObjectInput = (
                    await invigilator.async_answer_question()
                )
                if response.validated:
                    self.answers.add_answer(response=response, question=question)
                    self._cancel_skipped_questions(question)
                else:
                    breakpoint()
                    if (
                        hasattr(response, "exception_occurred")
                        and response.exception_occurred
                    ):
                        raise response.exception_occurred

These are all design problems. The proximate cause is that the validator for the question is not expanded:

(Pdb) response.exception_occurred
QuestionAnswerValidationError(QuestionAnswerValidationError(1 validation error for ChoiceResponse
answer
  Input should be '{{ answer_translation.answer[0] }}', '{{ answer_translation.answer[1] }}', '{{ answer_translation.answer[2] }}' or '{{ answer_translation.answer[3] }}' [type=literal_error, input_value='Unfair', input_type=str]
    For further information visit https://errors.pydantic.dev/2.9/v/literal_error))
(Pdb) 

what this means is that the construct validator needs to take a current_answers parameter to populate these fields so the validator works.

Originally posted by @johnjosephhorton in https://github.com/expectedparrot/edsl/issues/1107#issuecomment-2386060493

johnjosephhorton avatar Oct 03 '24 08:10 johnjosephhorton