New status: "not processed"

[pdawyndt]

trafficstars

Dodona might support a status "not processed" (incorrect) to indicate that a scheduled test has not been executed, e.g. in case previous testcases have timed out, or if a critical error has occurred.

This way, a judge might report on all scheduled tests, even if some of these tests have never been executed. Avoids that users might assume that all tests have succeeded except for the last one, in case of a timeout.

[bmesuere]

Before implementing this, we would suggest that a judge that wants to do this renders the "unprocessed" tests as false, but adds a message that the test was not executed. If we see that this is a common use case, we can then add support for it through a new status.

[chvp]

The JavaScript judge already hacks around this not being supported. I'm not necessarily arguing that this should be added, but it is already a thing that happens.

[niknetniko]

Before implementing this, we would suggest that a judge that wants to do this renders the "unprocessed" tests as false, but adds a message that the test was not executed.

TESTed does exactly this at the moment.