Calculate an exercise's difficulty based on other students' performances

[felipecalvo]

This would be useful for providing an answer to the question 'how hard is this exercise?', specifically thinking of flow navigation.

A couple things to take into account:

• A manual difficulty should still be set by the content creator, as initially there'd be no data with which to calculate anything. There should be no more than a couple options - think easy, medium, hard, very hard - and is not expected to be very precise, firstly because it'll be replaced with an automatic value later on, and secondly because the content creator's bias and previous knowledge may make them choose an incorrect difficulty
• This should be cached somehow and be updated periodically, maybe with a cron, as the query would be too expensive to be done on demand
• Some changes in an exercise's content could trigger a recalculation. For example, adding more tests could make it harder, or adding a tip could make it easier. (It could very well be a spelling fix, which wouldn't affect the difficulty). In any case, no change should be determinant enough as to make a significant variation in difficulty.
  • Even if the exercise is now pronouncedly easier or harder, this difficulty offset could fix itself in the long term if we use a time window, for example, taking into account assignments sent in the last six months only.
• We should consider what's the most convenient statistic: average, median, etc
• Cheating students and million-submissions-per-exercise students should be considered to avoid distorting the real difficulty
• Reading exercises and similar could be fixed at difficulty 0, as they can't be failed and there's no point in wasting resources in them