cms icon indicating copy to clipboard operation
cms copied to clipboard

Allow setting separate time limit for interpreted languages

Open prandla opened this issue 2 years ago • 2 comments

This allows setting a different (presumably higher) per-task time limit for some languages. This is mainly intended to make competing in Python viable without giving C++ users an advantage.

If a task doesn't have the interpreted language time limit set, the default time limit will be used instead. The set of interpreted languages is currently hardcoded as Python (2/3) and PHP.

I made this as a pull request directly because it's what we already use for Estonian olympiads, but admittedly it's not as configurable as it could be (mainly the list of which languages count as "interpreted"). The set of interpreted languages needs to be carefully chosen to make sure all such languages have roughly the same performance, otherwise competing in one language might be impossible, or in another language asymptotically bad solutions might get through the tests. Realistically it should be all of the slow languages that the jury has written example solutions for, with the time limit set accordingly to make all of them pass.

This makes me think that perhaps the set of languages should be a contest-specific setting, or at the very least something in cms.conf. But feedback is of course welcome.

prandla avatar Nov 19 '23 20:11 prandla

Admitting that different solutions could be subject to different time/memory limits opens a whole can of worms :-)

First, there are fairness issues (described to some extent in my paper on security of grading systems). But I agree that it's not a significant problem in some contests (e.g., first rounds solved at home).

Second, once you admit the possibility, it is no longer clear that it makes sense to have just two categories of languages and that the only difference should be in time limits. Perhaps you can also set a different memory limit for Java solutions? Or to give C# and Java a little more time, but as much as to Python?

I think it would be much better (even though more complicated) to allow setting of limits per language in every task.

gollux avatar Nov 29 '23 16:11 gollux

I am aware of the fairness issues, I tried that (embedding C code in Python using ctypes) once myself and only got points deducted after the end of a week-long contest, which i am still a little salty about :) However, in my opinion, clearly forbidding such schemes in the contest rules would be enough.

I agree that setting per-language limits would be better. However, I haven't thought of any good way to show these in CWS. The "task overview" page lists each task as a single row, so if there's more than 2 sets of limits I think it'd get quite unwieldy. An individual task's statement page has a separate row for the time limit, this could either become multiple rows or have something like "C++: 1 sec, Python: 3 sec" as the value. And yes, memory limit should be configurable per-language too if we're already going that far.

prandla avatar Nov 29 '23 17:11 prandla