Write proper integration tests

[hasufell]

I'm envisioning:

• run in docker, because they're gonna be stateful (no problem with hspec)
• use golden tests and check file-trees after each command

Also review prior bugs to get ideas about what to test (gitlab CI script also does that partly).

The basic idea is to compare file trees of ~/.ghcup before and after certain steps (e.g. installation of a tool). This is the simplest form of test that should exist for all commands (install, test, rm, ...) and run very isolated with a clean empty ~/.ghcup directory.

Then the next phase would be making sure that certain commands don't accidentally interfere (this happened)... e.g. that 'ghcup rm hls 1.5.0.0' doesn't delete a GHC version.

Further, some commands are more peculiar to test (e.g. 'ghcup run' or 'ghcup whereis').

[dfordivam]

I thought about how this could be done.

We could start with simple bash scripting to run commands in docker, and test for existence of files/dirs. This approach would be easy to run locally and on CI. But its could be rather difficult to write/maintain the bash script(s) with all the testing logic.

In order to scale this to a proper "testsuite" I looked around, and found this as an inspiration https://github.com/GoogleContainerTools/container-structure-test

This is a nice tool which almost does what we want. But it does not allow stateful operations. ( the docker container is immutable)

But their specification of tests in YAML looks good, and this style of writing test is much better.

So YAML + a simple test driver looks like like a good approach. Now the test driver could in principle be a bash script, or a tiny Haskell app with yaml parser is also possible.

[dfordivam]

On a second thought, I think I misunderstood the Github CI compatibility of the script+docker idea.

Ideally we want to run the tests locally and on GH actions. For local testing it makes sense to use the docker for doing all the ghcup commands inside the container. But this testsuite/script wont work as-it-is on the GH actions. (As it is not possible to run the regular docker commands on GH, which would typically mean running a container inside a container.) When run on GH actions, it makes sense to run the commands in the GH container directly, without invoking docker.

Also I think it would be ok to use hspec for these tests. For local run the tests would invoke docker, for GH action everything would be run directly.

[hasufell]

• the test suite itself should not invoke docker: if you wish, you should be able to run the tests on your bare filesystem
  • tests are meant to be executed on the end users system (e.g. after ghcup installation)
  • docker is overkill for local test execution on the end users system, we should support running the test suite through bubblewrap... if it exists (ghcup test is a thing)
• execution would be running somewhere in /tmp (through setting GHCUP_INSTALL_BASE_PREFIX)
• I don't want to depend on frameworks... all we need is a way to compare file trees (e.g. through find) and do minimal toolchain invocations (e.g. ghc can compile a hello world)
• if we keep using shell, then it must be posix, not bash
• it needs to be packaged somehow, so that ghcup test can download and execute it