Add CI to ensure that exercise tests are supported by the test runner
Fuzz testing is great, but this is not handled yet by the code in the test runner that extracts the test code to show it to a user of the online editor. So we should make sure that Fuzz testing is not introduced unintentionally.
@ceddlyburge I agree, we should also check (if possible) that test descriptions are string literals.
Ideally it would be good to actually run the tests runner on the exercise tests during, which would make sure that they work. This might be tricky though, being as the test runner lives in another repo.
At the moment we are aware of two cases that the test runner can't handle (although there are probably more), so it might be easier to just check for these instead
- Fuzz tests
- Anything other than a string literal for the test description
We should also prevent any expression creating context outside of a test expression like:
let context = "value not visible in the extracted snippet"
in test "description" <|
\_ -> Expect.equal -- something using the variable `context`
Because otherwise, the snippet presented to the student will contain usage of the context variable, but the student will not know what is the actual value of that variable.
Ideally it would be good to actually run the tests runner on the exercise tests during, which would make sure that they work. This might be tricky though, being as the test runner lives in another repo.
TIL it's actually not tricky at all 😸
It would be great to do here before attempting to solve this issue.