Test test functions with wrong solutions

[lowyck]

trafficstars

When I define an exercise with some particular conditions for success, I’m not sure that my test functions correspond to what I want to do. For example, I have an exercise asking to sort a list in linear time. To be sure that my measurement is correct, I want to define and grade several answers for which I know what is false.

I think of it this way: You add files wrong_solution_{1..n}.ml to the exercise. They are different versions of solution.ml, with a few mistakes. In our example, a function that is not linear in time. When building or grading, their essays (the wrong_solutions) are graded. They should not get full points. Otherwise, an error is diplayed. There can also be an optional file wrong_solutions.expected_points with some syntax describing the points awaited for the essays (or a range).

This suggestion is a follow-up of #58 . I’m opening it as an issue on Learn-OCaml because I think the automatic grading of wrong solutions should be integrated in the grader and/or build and activated in case there are files named wrong_solution_{1..n}.

[AltGr]

That would indeed be nice: you can at the moment test manually, or each wrong solution of each exercise, but integrating that into the build would be useful.

Note that the learn-ocaml grade interface has improved a bit since #58: the results and score are now printed by default!

So the functionality would be roughly that of this script:

set -e
for wrong in $EXERCISE_DIR/wrong_solution_*.ml;  do
  ! learn-ocaml grade -e $EXERCISE_DIR -s $wrong
done

Note that, for this, we should probably disable the random input part of all expect tests, so that you are sure to catch the failures you want to no matter what the sampler gives.