links
links copied to clipboard
Make use of expect tests (or similar)
This is an issue to track improving the Links testing structure. Currently Links mainly makes use of an ad hoc test harness which performs the equivalent of a "system test" (i.e. testing the entire Links executable on a single snippet of code).
An alternative approach could be to use something like expect tests which can then also be extended to support unit tests. Expect tests are tests which execute some code and compare the standard output / error output of the code snippet to a correct version. If the output is not exactly the same a diff is provided which can then be accepted by running dune promote
.
There are some advantages to this approach:
- Expect tests are used quite commonly, making them well supported
- Writing unit tests are helpful to the programmer. They can illustrate how to use core components of the compiler and can demonstrate special behaviour. They are also an easier way to document code than writing actual documentation.
- I assume that in some form or another expect tests are / can be run in parallel. Currently our CI is quite slow so any bit to get results quicker can be nice.
- Tests could be written in a way that they don't need to spawn a new Links executable on each run. This could potentially make tests run much faster.
- Expect tests in particular encourage the programmer to provide pretty printers for all data types. Having access pretty printers everywhere then makes it easier to debug code.
- Expect tests encourage making each software component more user friendly in a standalone setting.
- Existing OUnit tests could relatively easily be ported to expect tests, unifying all of our testing framework.
Pull request #883 demonstrates how the existing test harness can fairly easily be ported to expect tests.