Vision for organizing tests

[kyouko-taiga]

I want to open this issue to share my vision about the future of testing for Val compiler.

First, I want to distinguish between 3 categories of tests:

1. Unit tests of Swift components
2. Integration tests of compiler parts
3. End-to-end tests of compiled/interpreted Val programs.

Unit tests should obviously be written in Swift and organized according to the common practice. For integeration and end-to-end tests, I want to favor writing tests in Val rather than having Val fragments scattered across Swift files. I believe inline Val code should be used mostly for the purpose of unit testing components that require an AST.

Writing separate Val files will help us progressively build a corpus of programs that I believe to be valuable for a handful of reasons:

1. to provide a collection of Val program examples that people can find easily.
2. to provide an easy way for people to contribute new tests without having to know anything about Swift.
3. to facilitate looking for recurring patterns in Val code with classical grep-like approaches.
4. to facilitate reusing test cases.

For the sake of simplicity, the proposed architecture currently assumes a program to be a single Val file. However, we should keep in mind that we'll eventually have to write tests for programs consisting of several files or even modules. That being said, I believe having a great setup for handling individual files is a clear step toward one that can handle multiple modules.

I want to avoid (without being completely opposed to) "hacking" features into Val for the purpose of testing it from Swift. Testing is an important part the developer's experience so we should be sure to develop a satisfying story in Val, and one that stands on its own. I believe having to eat our own dog food will help us making informed decisions.

Although building such a test framework in Val comes is quite ambitious, I believe we can do it progressively. We would start by simply running programs and check their exit status, quickly implement assertions to support more elaborate tests, and then slowly iterate on the design of a library.

Now let's have a look at the proposed arborescence of Tests/:

Tests/
  |- CompilerTests/
  |  |- CompilerTests.swift
  |  |- TypeChecking.testsuite/
  |  |- CXXCodeGen.testsuite/
  |  |- LLVMCodeGen.testsuite/
  |  |- Parsing.testsuite/
  |  |  |- UnterminatedComment.val
  |  |  |- ...
  |
  |- UnitTests/
  |  |- Core/
  |  |  |- SourceFileTests.swift
  |  |  |- ...
  |  |
  |  |- Utils/
  |     |- ...
  |
  |- Val
  |  |- ValTests.swift
  |  |- Foo.testsuite/
  |  |- Bar.testsuite/
  |  |- Library/
  |  |  |- Core.testsuite/
  |  |     |- IntTests.val
  |  |     |- ...
  |
  |- Utils/
  |  |- SomeUsefulComponent.swift
  |  |- ...

CompilerTests/

This directory would contain integration tests.

A single test would consist of running one of the Val programs in one of the test suites. For example, UnterminatedComment.val would be a Val source file with test annotations, just like we have now.

CompilerTests.swift would implement the test runner for all of these tests, taking care of the integration with XCTest.

Val/

This directory would contain end-to-end tests.

These tests would not use the annotations. Instead, they would be written as regular Val tests in the framework we'll be developing for Val, i.e. starting with exit statuses before moving to assertions and eventually something more sophisticated

Tests in this directory would be eventually executed for our three backends: C++, LLVM, and the interpreter. ValTests.swift would implement the test runners and take care of the integration with XCTest.

UnitTests/

This directory would contain unit tests for the our individual Swift components. Each sub-directory would correspond to a target and contain a file for each tested component.

Nothing fancy here; just good old Swift unit tests.

Utils/

This directory would contain the Swift helper components shared across multiple test targets, should we need some.

Related issues: #403