WIP.: Systest Postmorten

[dsmello]

Motivation

Add postmorten handler on the systest to allow the devs to have time to inspeact the cluster after a fail.

Description

Test Plan

TODO

• [ ] Explain motivation or link existing issue(s)
• [ ] Test changes and document test plan
• [ ] Update documentation as needed
• [ ] Update changelog as needed

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 82.0%. Comparing base (787351f) to head (3b7c986). Report is 90 commits behind head on develop.

Additional details and impacted files

@@           Coverage Diff           @@
##           develop   #6222   +/-   ##
=======================================
  Coverage     82.0%   82.0%           
=======================================
  Files          308     308           
  Lines        33906   33906           
=======================================
+ Hits         27810   27812    +2     
+ Misses        4320    4318    -2     
  Partials      1776    1776           

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[fasmat]

This is the code that deletes the namespace after a test:

https://github.com/spacemeshos/go-spacemesh/blob/787351f6a789c0446b37acf27aecc2a7afd1343d/systest/testcontext/context.go#L369-L377

So if you want that test to stick around you can either ensure that cctx.Keep is true by passing the keep flag to the test runner, or if you only want that to happen when a test fails adjust the cleanup function to only execute the Namespace deletion when the test passes:

func cleanup(tb testing.TB, f func()) {
	tb.Cleanup(func() {
		if !tb.Failed() { // only execute f() when test passed
			f()
		}
	})
	signals := make(chan os.Signal, 1)
	signal.Notify(signals, syscall.SIGINT, syscall.SIGTERM)
	go func() {
		<-signals
		f()
		os.Exit(1)
	}()
}

[acud]

@dsmello I'm not sure this is the way we wanna go. It adds a lot of complexity to the way we run the tests and this will make our lives more difficult in the future if we'd like to refactor this stuff away, since we will be relying on the functionality that it provides.

@fasmat has some other ideas on how to execute on the idea and provide the same functionality. Maybe we could all jump on a call these days and discuss this stuff?