Adaptive grading

[Xyene]

Adaptive grading should be added to the judge to support more interesting problems, as well as "weird" IOI-level problems. A good implementation should modify as little of the existing framework as possible.

After giving it some thought, I've come to the conclusion that the only clean way to go about this is have judges entirely in control of updating scores. That is, a judge should be able to, at any time, send a packet to change the score of an old submission — actually regrading all solutions every submission for an adaptive problem is impractical, as is making the site flexible enough for all adaptive problem types, so the judges must be entrusted with sending score updates for these problems.

The way I see this playing out is as an extension of custom graders:

• The addition of a create_persistent_file in judgeenv that creates a file in a R/W directory that persists across judge reboots. Options:
  • A new judge config element
  • Default to the problem's data folder
  • For our setup, the first option is best (set up a R/W NFS mount), while the second should be supported for others
• Graders create e.g. an SQLite database from that file, and store submission id - case - score - extra param data in it. I see graders executing queries of the form: SELECT TOP 1 submission_id, MAX(score) as mx GROUP BY score ORDER BY mx
• The addition of a send_rescore_packet taking data of the form:

{
   submission_0_id: {
       case_0_score: ...,
       case_1_score: ...,
    },
    ...
}

Bridge-side, this would be easy to add support for.

Concerns

• Judges would be able to rescore contest submissions, and this would likely happen after the first out-of-contest submission. This is very bad; we can either send a flag specifying a submission is in-contest (and hope the graders flag it differently — prone to problemsetter error) or implement DMOJ/site#128. This isn't a show-stopper for this ticket, but it is one before adaptive problems can be used in contests.
• This gives a compromised judge the ability to arbitrarily rescore submissions. Is this really an issue, considering a rejudge should be deterministic? Probably not.
• Interacting with the proposed HTTP data server — interaction is not needed, we can just run all adaptive problems on one or two judges that are actually hooked up with NFS.
• Can we lock a SQLite table easily? If not we can always have graders use something more heavy-duty like MySQL, or restrict adaptive problems to a single judge. This is more of a concern for grader implementation rather than support for adaptive grading.
• Size of judge-side storage — very likely insignificant even for thousands of submissions.