cms
cms copied to clipboard
Improve evaluation messages
There are two main problems with the current evaluation messages:
- For the "Execution killed" message, exceeding the memory limit is by far not the most common cause of this outcome. The much more common case of undefined behavior/crash is not mentioned.
- For the "non-zero exit code" message, the most common cause is from an unhandled exception being thrown in a language such as Java or Python. But the help text doesn't mention this at all, and the contestant is confused. (We get a lot of questions about this in our national olympiad.)
This pull request improves these evaluation messages and help texts to match the most common causes. For the "Execution killed" message, I picked the "runtime error" hint as a concise and easily understandable variant. The "wall clock limit exceeded" message help text is also updated.
These changes should be discussed and possibly improved — in particular the evaluation messages themselves, as they are mentioned in IOI contest rules. Perhaps somebody from ISC/ITC should check these.
In some natural languages, the words "execution" and "runtime" will translate to the same word. So one optional thing to consider is simplifying "Execution killed (runtime error)" to just "Runtime error". Or rephrasing this text in some other way. On the other hand, this message should match the similar compilation outcome.
Codecov Report
Merging #1134 into master will decrease coverage by
<.01%
. The diff coverage is70.83%
.
@@ Coverage Diff @@
## master #1134 +/- ##
==========================================
- Coverage 61.99% 61.98% -0.01%
==========================================
Files 230 231 +1
Lines 16604 16627 +23
==========================================
+ Hits 10294 10307 +13
- Misses 6310 6320 +10
Flag | Coverage Δ | |
---|---|---|
#functionaltests | 45.61% <4.16%> (+0.01%) |
:arrow_up: |
#unittests | 43.33% <70.83%> (+0.04%) |
:arrow_up: |
Impacted Files | Coverage Δ | |
---|---|---|
cms/grading/steps/evaluation.py | 78.04% <ø> (ø) |
:arrow_up: |
cms/db/__init__.py | 96.42% <100%> (ø) |
:arrow_up: |
cmscontrib/updaters/update_43.py | 69.56% <69.56%> (ø) |
|
cms/grading/Job.py | 83.41% <0%> (-5.69%) |
:arrow_down: |
cms/service/ProxyService.py | 57.44% <0%> (-1.6%) |
:arrow_down: |
cms/db/usertest.py | 95% <0%> (-0.72%) |
:arrow_down: |
cms/service/EvaluationService.py | 67.55% <0%> (ø) |
:arrow_up: |
cms/db/filecacher.py | 77.37% <0%> (+0.32%) |
:arrow_up: |
cms/server/admin/handlers/base.py | 68.13% <0%> (+0.33%) |
:arrow_up: |
... and 5 more |
Continue to review full report at Codecov.
Legend - Click here to learn more
Δ = absolute <relative> (impact)
,ø = not affected
,? = missing data
Powered by Codecov. Last update d4c9e92...dbb71ad. Read the comment docs.