elm icon indicating copy to clipboard operation
elm copied to clipboard

Add CI to ensure that exercise tests are supported by the test runner

Open mpizenberg opened this issue 4 years ago • 3 comments

Fuzz testing is great, but this is not handled yet by the code in the test runner that extracts the test code to show it to a user of the online editor. So we should make sure that Fuzz testing is not introduced unintentionally.

mpizenberg avatar Sep 07 '21 13:09 mpizenberg

@ceddlyburge I agree, we should also check (if possible) that test descriptions are string literals.

mpizenberg avatar Mar 26 '22 16:03 mpizenberg

Ideally it would be good to actually run the tests runner on the exercise tests during, which would make sure that they work. This might be tricky though, being as the test runner lives in another repo.

At the moment we are aware of two cases that the test runner can't handle (although there are probably more), so it might be easier to just check for these instead

  • Fuzz tests
  • Anything other than a string literal for the test description

ceddlyburge avatar Mar 27 '22 09:03 ceddlyburge

We should also prevent any expression creating context outside of a test expression like:

let context = "value not visible in the extracted snippet"
in test "description" <|
  \_ -> Expect.equal -- something using the variable `context`

Because otherwise, the snippet presented to the student will contain usage of the context variable, but the student will not know what is the actual value of that variable.

mpizenberg avatar Apr 11 '22 11:04 mpizenberg

Ideally it would be good to actually run the tests runner on the exercise tests during, which would make sure that they work. This might be tricky though, being as the test runner lives in another repo.

TIL it's actually not tricky at all 😸

It would be great to do here before attempting to solve this issue.

jiegillet avatar Nov 09 '22 14:11 jiegillet