Testing suite roadmap

[ghost]

There has been great progress in building up the test part of the test suite, however the software running the tests is quite lacking. A replacement should be developed that fulfills the following criteria.

• [x] imports src/main.nim instead of running pre-built binaries
• [x] Runs tests parallel (multiple instances of japl)
• [x] Separates stdout/stderr output

With this, a more advanced testing language is needed.

• [x] Replace output: with stdout: stderr: (and stdin:)
• [x] Allow for multiple tests per file
• [x] Allow regex (or peg) matches (handy for exception messages)

The visual of the test runner also needs improvement:

• [x] Add a progress bar indicating test progress
• [x] Improve the CLI (flags for verbosity and such)

Long term improvements:

• [ ] Allow writing a parallel test in python (checks if output matches)
• [ ] Benchmark support

[ghost]

The following issues came up, which need fixing: Bugfixes:

• [x] Investigate stdin crashing
• [x] rename stderrre and stdoutre to matchout and matcherr
• [ ] finish documentation
• [ ] move localization to a centralized place
• [x] customizable kill time

edit: opened one of them in its issue