Turn on and off a node

[GalRogozinski]

Description

Simply turn off a node after a certain amount of time

Motivation

If the node manages to turn itself off without an error then all is fine, else we have a regression. Currently Travis finds those issues and aloe needs to do it as well

Issues / Scenarios

Node can be turned on and off without an error

How to test for this

Just sleep for (X). We already have a step in the pipeline that will later check for errors.

[kwek20]

Rarely a node shuts down with a SEGFAULT from rocksdb, how would we handle this? As of yet i have no idea what is causing this....

[GalRogozinski]

I didn't look at the core reason but it seems like rocksdb doesn't like being killed while writing to the db... shocking I know

[kwek20]

This also happens with a gracefull shutdown, that should wait for pending ops right?

[GalRogozinski]

If we wait long enough it doesn't We can start the test with a long sleep and decrease it after #1576 is done