Feature request: multi-part exercises

[adhameer]

It would be extremely useful to be able to organize an exercise into sub-parts, perhaps similarly in layout to the tutorials where you can navigate through the sections using left and right arrows at the top of the screen.

Each sub-part would have its own prelude, prepare, template/editor, description, and solution. They could all share a toplevel (though there might be some issues to work out here wrt prelude/prepare definitions).

This would be very helpful because often later parts of an exercise will rely on successfully implementing the earlier functions. For example, part a) of an exercise will require the student to implement a function f, and then parts b) and c) involve making use of f. If a student is unable to implement f, they will be harshly penalized by the autograder on parts b) and c) of the exercise as well. In contrast, when manually grading an assignment we could still grade the student on parts b) and c) independently, if they are still able to use f properly assuming a correct implementation.

This is especially a problem with exercises involving modules. In a recent exercise, we wanted students to implement a module M with an abstract type t, and later wanted them to implement a functor F (M): S with type t = M.t (where S was some signature) and apply it to M resulting in a module M2. We were unable to test the module M2 since its type involved some types defined by the student in their solution, so we were not able to provide this type in our grader code.

Splitting the assignment up into several exercises, each with the previous questions implemented in the prepare.ml file, would have been a way to get around this, but we felt that doing this was a bit clunky and confusing. Being able to split up an exercise into sub-exercises which can be navigated between on a single page seems like a good solution to this.

[erikmd]

Dear @adhameer, @hannelita, @yurug, and all people interested in this feature,

FYI this use case (multi-part exercises) appears to be quite close to the use case #395 (dual-grader exercise), so @YoanwM and I are considering to devise an implementation that addresses both use cases:

• with the "simplest" possible design choices to address them in a natural way (while being extensible);
• in the most backward-compatible way with the "exodir" specification of learn-ocaml 0.12.

So we may post in this issue (or in #395) more details in the upcoming weeks, but the overall idea is as follows:

• Considering that the current "exodir" spec allows nested sub-directories in exercises/index.json;
• We'll keep the structure of the sub-exercises (each being a standard exo dir: exo1/part1/{descr.md, test.ml, …}),
• But the exo1/part1/meta.json won't be taken into account,
• Instead, we'll add an exo1/subindex.json file that will specify in particular:
  • Some basic metadata about the whole exercise exo1 (difficulty, etc.)
  • The name and the order of the sub-parts
  • Whether some sub-parts should be hidden to students by default (typically to address #395)
  • What is the scoring strategy:
    1. relying on weights (some integer ≥ 0 for each sub-part → using scaling coefficients, rather than percentages)
    2. or relying on the note of a single part (typically if the last grader only matters for #395), but this 2nd strategy could just be emulated with the 1st one, if only one part has a non-zero weight.
  • Whether each partn/solution.ml should also be checked (at learn-ocaml build time), with the previous partn-1/test.ml → which makes a lot of sense if the successive parts are tightly related, or even are a refinement of the grader for the same exercise; but this setting would be optional − and enabling it for each subpart would just induce a O(2⋅n)−like slowdown for learn-ocaml build [not O(n²)].

[erikmd]

(As an aside, the runnning example (exercises involving modules) in @adhameer's post does not necessarily require some multi-part exercise support: it appears we can implement a single grader with independent tests, for dependent modules/functors… but indeed, it'll be much more natural to benefit from this multi-part feature to achieve this :)

[erikmd]

Hi @yurug @YoanwM, following our last video meeting and for the record, here is the recap of the format we finally proposed for subindex.json :relieved:

Grammar of subindex.json:

<directory> = <string>

<meta> = {
  "kind"               : ( "exercise" | "problem" | "project" ),
  "stars"              : [1 .. 5],
  "title"              : "Title of the multi-part exercise",
  /* In an exercise repository, each exercise must have a unique identifier. */
  "identifier"         : "some_unique_identifier",
  /* Authors with their emails */
  "authors"            : [["First Last", "[email protected]"], ...],
  /* The skills and concepts that are practiced by this exercise. */
  "focus"              : ["skill1", ..., "skillN", ..., "concept1", ..., "conceptM"],
  /* The skills and concepts that are required to do this exercise. */
  "requirements"       : ["skill1", ..., "skillN", ..., "concept1", ..., "conceptM"],
  /* The suggested exercises in case of success, RELATIVE TO THE REPO ROOT */
  "forward_exercises"  : [ "exercise1", "exercise2", ... ],
  /* The suggested exercises in case of difficulty, RELATIVE TO THE REPO ROOT */
  "backward_exercises" : [ "exercise1", "exercise2", ... ] }

<part> = {
  "subtitle": <string>,
  /* Path to the exo, RELATIVE TO THE SUBINDEX */
  "subexercise": <directory>,
  "student_hidden": <boolean>, /* Optional: false by default */
  "student_weight": <integer>,
  "teacher_weight": <integer> }

{ "learnocaml_version": <string>,
  "meta": <meta>,
  /* Path to the ref. grader exo for retrograding, RELATIVE TO THE SUBINDEX */
  "check_all_against": <directory>, /* Optional: no retrograding by default */
  /* Last but not least: */
  "parts": [ ( <part> )+ ] }

Complete minimal example of subindex.json w.r.t. the use case of #395:

{ "learnocaml_version": "1",
  "meta": {
    "learnocaml_version": "2",
    "kind": "problem",
    "stars": 4,
    "title": "PFITA - DM1",
    "identifier": "pfita-dm1",
    "authors": [["Yoan Mollet", "[email protected]"], ["Erik Martin-Dorel", "[email protected]"]],
    "focus": ["recursion"],
    "requirements": ["lists", "pattern-matching"],
    "forward_exercises": [],
    "backward_exercises": [] },
  "check_all_against": "dm1-prof",
  "parts": [
  { "subtitle": "First tests",
    "subexercise": "dm1-etu",
    "student_weight": 1,
    "teacher_weight": 0
  },
  { "subtitle": "Grading",
    "subexercise": "dm1-prof",
    "student_hidden": true,
    "student_weight": 0,
    "teacher_weight": 1
  }]
}