runner
runner copied to clipboard
Improve output of tests for frameworks which have no `<IT::>`
Initially asked on Discord:
Codewars runner supports some languages and testing frameworks which do not fit well into the BDD pattern and do not have constructs which map well into <DESCRIBE::> and <IT::> markers used by test output panel. As a consequence, some reporters emit artificially created <IT::> tags, with titles derived from some additional context data because authors cannot assign any title to them. This often results in an awkward output with confusing or misplaced titles (see #239 , #241 , https://github.com/codewars/FactCheck.jl/pull/3 ), or which spam <IT::> beyond reason.
My proposal would be to get rid of artificial <IT::>'s from all reporters for frameworks which do not have a corresponding construct, and add support for assertions which are not in scope of any <IT::>. Maybe it would be good enough to allow for assertions inside of <DESCRIBE::>, or maybe a better solution would be to provide some new marker, which would be used for frameworks which do not map well into BDD and use simpler concept of a test and a context, for exampel something like:
<TEST::>
<TEST::> (if nesting is allowed)
<ASSERTIONS_CONTEXT::> (optional)
<ASSERTIONS_CONTEXT::> (if nesting is possible)
<ASSERTION::> (probably <PASSED::> or <FAILED::>)
<ASSERTION::> (if not stop on first failure)
<COMPLETEDIN::>
<COMPLETEDIN::>
<COMPLETEDIN::>
<COMPLETEDIN::>
With such setup, Julia reporter would emit <TEST::> for @facts (which is optional), <ASSERTIONS_CONTEXT::> for @context (which is optional), and <PASSED::> or <FAILED::> for @fact. For Clojure, we'd get <TEST::> for deftest (whici is mandatory), <ASSERTIONS_CONTEXT::> for testing (which is optional), and <PASSED::> or <FAILED::> for is.
For what it's worth I admit that the main reason behind this ticket is the fact how discouraged I can sometimes be to fix some translations, because no matter what I try, there's simply no way to get a nice and clear output of tests for some languages. This can be also a problem for reviewers who can be not familiar with intricacies of setup of some languages on Codewars and sometimes request to make the output of tests clearer before a translation gets approved, even though it's not possible/not easy.
Another example: Perl
The test 1 blocks have no corresponding construct in code of tests and are most probably emitted by CW reporter to provide an obligatory <IT::>. It is not known how to provide a meaningful title for such it instead of the test N.
Another affected language: Raku
is this related to why printing anything in Perl/Raku seriously messes with the output panel ?