Custom hooks for running commands before/after test binary

[dbrgn]

I have a project with a lot of integration tests written in Python. These tests run against the main binary.

With --command build I can launch the test server through tarpaulin, then I can manually run the test.py script, and then tarpaulin collects the coverage (as long as the test script has a way to gracefully stop the test server). This works fine. However, I have to manually start both tarpaulin and the test script.

It would be nice if tarpaulin would support hooks:

• before_start would be run before the target binary (i.e. the test binary or the main binary) is started. This could be used for things like starting a service required by the binary.
• after_start would be run in the background as soon as the target binary has been started. When the target binary is a service, then this can be used to launch the integration test suite (e.g. /usr/bin/python test.py).
• after_end would be run after the target binary has terminated.

Those hooks could be defined as a list of shell commands. Each shell command is run in sequence. Additionally, it would be nice if certain placeholders could be passed to the commands. For example:

after_start = [
  "/usr/bin/env python3 test.py",
  "/usr/bin/kill -USR1 {pid}"
]

In the example above, the {pid} placeholder could be replaced with the target binary PID by tarpaulin. This would allow stopping the service when a USR1 signal is sent to it. Thinking further, the after_end task could support variables containing things like total coverage percentage.

Something we'd need to think about when implementing this feature would be the handling of stdout/stderr. If both the service and the test script print a lot of logs, those would be interleaved, which would be hard to read. Maybe tarpaulin could allow configuring the output target for both the target binary and every hook. The output could be sent either to stdout/stderr, or to a file. That would be very useful in CI, because the output from the test suite could be printed directly on stdout/stderr, while the output from the server could be sent to a log file, which could then be stored as a build artifact (for debugging in case something went wrong).