Overhaul test-suites

[fendor]

A single test-run takes a huge amount of time and, by now, we have a huge array of VM tests. Currently, we have the resulting matrix of test-runs:

Tool\GHC	8.4.2	8.4.3	8.4.4	8.6.1	8.6.2	8.6.3	8.6.4	8.6.5	8.8.1
Cabal	:x:	:x:	:heavy_check_mark:	:x:	:x:	:x:	:x:	:heavy_check_mark:	:heavy_check_mark:
Stack	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:

Resulting in 12 VMs to just build in every configuration, making it 36 VMs for every supported operating system: Linux | MacOs | Windows.

Tests take roughly 20 minutes or even more (on my machine 26 min) to run.

To make the matter worse, we should have tests where only either cabal or stack is installed and both.

What I want to discuss in this post is an overhaul of the test system with the following points:

• ~~Functional tests import suggestions are incredibly slow~~
  • Demote them to unit-tests # Fixed by #1591
• Prefer direct cradle over stack cradle
  • On my system a call to stack repl takes roughly 1.5s,
    • I estimate we have 140 calls to stack repl, replacing this with a direct cradle will reduce run-time by two minutes at least
• Add tests for loading project context with stack and cabal, similar to cabal-helper.
  • Since hie-bios abstracts the build-tools, we dont have to execute all tests for each build-tool as long as we test that the build-tools initialize the context correctly and provide compilation arguments.