Allow TestStep commands to be used in assert

[ANeumann82]

What would you like to be added: It would be great if we can use a command in an assert step that will be executed repeatedly until it returns true or the timeout happens and it fails.

Why is this needed: At lot of time things need to be asserted that may not be available as yaml:

For C* I had to fall back to use a bash script with my own timeout:

https://github.com/mesosphere/kudo-cassandra-operator/blob/master/kuttl-tests/suites/failure-recovery/node-failure/assert-node-up.sh

[kensipe]

thanks @ANeumann82 I thought this was a dup but I can't find it... this is a great idea!

Thoughts / Questions:

1. Are we saying that exit code 0 is true and all other codes are false? I'll work for now as if that is true
2. Should it always repeat for a period of time? or exec command and it is either true or false?
3. is the timeout for the length of a command (which is what it is today) or is it for the length of time for repeats (retries)
4. Do we want "repeating" for a duration of time or number of times?
5. we want a sleep period between exec calls?

[ANeumann82]

1. Yeah, I would think that's a pretty good standard that we should follow
2. Single Command Execution is already possible today - It's not in assert step, but a command that has "ignoreFailure: false" works as well. I think the repeat for a period of time is a key point in this request. It should behave similar to an assert of a yaml resource.
3. I think the timeout should be for the repeats, but a timeout for a single execution may be nice to have
4. Same, I would probably use "repeating for a duration of time" most often, but some people might want number of times. I think number of times could be left out for now and we can add it later
5. I don't think so, but it would be a nice to have feature if we could configure the "executionTimeout" for a single exec and the "sleepDuration".

I think as long as we use the existing "Command" object that allows a script to be called we should be fine

[kensipe]

Grouping a number of issues "themed" around commands to milestone v0.7.0. We should work on them in the same iteration. Thanks for context and feedback.