cms icon indicating copy to clipboard operation
cms copied to clipboard

Improve evaluation messages

Open andreyv opened this issue 4 years ago • 1 comments

There are two main problems with the current evaluation messages:

  1. For the "Execution killed" message, exceeding the memory limit is by far not the most common cause of this outcome. The much more common case of undefined behavior/crash is not mentioned.
  2. For the "non-zero exit code" message, the most common cause is from an unhandled exception being thrown in a language such as Java or Python. But the help text doesn't mention this at all, and the contestant is confused. (We get a lot of questions about this in our national olympiad.)

This pull request improves these evaluation messages and help texts to match the most common causes. For the "Execution killed" message, I picked the "runtime error" hint as a concise and easily understandable variant. The "wall clock limit exceeded" message help text is also updated.

These changes should be discussed and possibly improved — in particular the evaluation messages themselves, as they are mentioned in IOI contest rules. Perhaps somebody from ISC/ITC should check these.

In some natural languages, the words "execution" and "runtime" will translate to the same word. So one optional thing to consider is simplifying "Execution killed (runtime error)" to just "Runtime error". Or rephrasing this text in some other way. On the other hand, this message should match the similar compilation outcome.


This change is Reviewable

andreyv avatar Aug 28 '19 13:08 andreyv

Codecov Report

Merging #1134 into master will decrease coverage by <.01%. The diff coverage is 70.83%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1134      +/-   ##
==========================================
- Coverage   61.99%   61.98%   -0.01%     
==========================================
  Files         230      231       +1     
  Lines       16604    16627      +23     
==========================================
+ Hits        10294    10307      +13     
- Misses       6310     6320      +10
Flag Coverage Δ
#functionaltests 45.61% <4.16%> (+0.01%) :arrow_up:
#unittests 43.33% <70.83%> (+0.04%) :arrow_up:
Impacted Files Coverage Δ
cms/grading/steps/evaluation.py 78.04% <ø> (ø) :arrow_up:
cms/db/__init__.py 96.42% <100%> (ø) :arrow_up:
cmscontrib/updaters/update_43.py 69.56% <69.56%> (ø)
cms/grading/Job.py 83.41% <0%> (-5.69%) :arrow_down:
cms/service/ProxyService.py 57.44% <0%> (-1.6%) :arrow_down:
cms/db/usertest.py 95% <0%> (-0.72%) :arrow_down:
cms/service/EvaluationService.py 67.55% <0%> (ø) :arrow_up:
cms/db/filecacher.py 77.37% <0%> (+0.32%) :arrow_up:
cms/server/admin/handlers/base.py 68.13% <0%> (+0.33%) :arrow_up:
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update d4c9e92...dbb71ad. Read the comment docs.

codecov[bot] avatar Aug 28 '19 18:08 codecov[bot]