enhancements to how kola handles certain failure scenarios

[miabbott]

From https://github.com/openshift/os/pull/685#discussion_r770681213

I'm thinking we should extend kola to handle these kinds of checks so they're non-fatal but still loud. Some ideas:

• have kola check test output for KOLAWARNING: ... and reprint the warning on the console.
• add a warn-on-rc: N field to kola test JSON where if a test exits with the given code, kola prints WARN: $testname instead of FAIL: $testname.

We can then have kola e.g. touch a file or print a report of warnings and the Slack notifier in the pipelines can check for this file and tweak the notification.

cc: @jlebon @dustymabe

[dustymabe]

IOW there are things that we want to get notified about without having the pipeline fail (and cause our momentum to get slowed).

I think this sounds like a good idea, but also think if enough warnings start coming our way we'll just ignore them like we do everything else that doesn't require action in order to get unblocked. I'm on the fence, but lean "no".

[jlebon]

I think this sounds like a good idea, but also think if enough warnings start coming our way we'll just ignore them like we do everything else that doesn't require action in order to get unblocked.

Being able to ignore warnings when we need to is valuable though. :) Ignoring it forever is a risk, but I think there's a low chance of that if it's loud enough in Slack/IRC. And most of those things by design are trivial to fix (e.g. just dropping an overlay), so it should be possible for anyone to do.