Output: Differentiate Soft-Fail From Hard-Fail

[AjkayAlan]

The Problem

Currently when running Checkov and getting output, you cannot tell what checks are hard-failing versus what checks are soft-failing.

This makes it notably difficult for end users to understand which failures are actually breaking their pipeline (when using Checkov in a pipeline) when a central team is managing a common config (to dictate what checks are hard-fail and what checks are soft-fail) which is abstracted from the end users (though those users can view the config if needed).

Example:

In the above image, I ran checkov against an offending Cfn template, with the args --soft-fail-on CKV_AWS_111 --quiet --compact. So my pipeline gets a nonzero exit code and CKV_AWS_33 is the only rule causing me to have a non-zero exit code.

Why It Matters

If it was clearer in output if a check is soft failing, it would allow end users to more easily troubleshoot and understand which rules they should prioritize fixing first.

Potential Solutions (likely a combination of some of the items stated below)

1. Change the color of the output of soft-failing checks from red to yellow or some other color to help indicate that the resource is still in violation of the rule, but it is not causing a non-zero exit code. Could likely do it at https://github.com/bridgecrewio/checkov/blob/05e61f5a6619bd4e3090625f4fc0be8bd82929dc/checkov/common/output/record.py#L137, though some more information would likely have to be plumbed in to the class.
2. Change the output text to better indicate that the check is soft-failed. Could likely do it at https://github.com/bridgecrewio/checkov/blob/05e61f5a6619bd4e3090625f4fc0be8bd82929dc/checkov/common/output/record.py#L192, or somewhere around those parts, but still would need more information sent to the class before you could do it
3. Add a SOFT_FAILED entry to the CheckResult enum, and some logic to shift FAILED to SOFT_FAILED based on if the check is in the soft_failed list. This one could be pretty painful based on the amount of checks to see if CheckResult == FAILED.
4. Something else?

[stale[bot]]

Thanks for contributing to Checkov! We've automatically marked this issue as stale to keep our issues list tidy, because it has not had any activity for 6 months. It will be closed in 14 days if no further activity occurs. Commenting on this issue will remove the stale tag. If you want to talk through the issue or help us understand the priority and context, feel free to add a comment or join us in the Checkov slack channel at https://slack.bridgecrew.io Thanks!

[AjkayAlan]

Bumping so this doesn’t auto close

[davidfischer-ch]

This is a good idea as I was investigating the pipeline result and cannot differentiate admitted failures from actual failures.

[stale[bot]]

Thanks for contributing to Checkov! We've automatically marked this issue as stale to keep our issues list tidy, because it has not had any activity for 6 months. It will be closed in 14 days if no further activity occurs. Commenting on this issue will remove the stale tag. If you want to talk through the issue or help us understand the priority and context, feel free to add a comment or join us in the Checkov slack channel at https://slack.bridgecrew.io Thanks!

[stale[bot]]

Closing issue due to inactivity. If you feel this is in error, please re-open, or reach out to the community via slack: https://slack.bridgecrew.io Thanks!

[pingoleon108]

Bumping so this doesn’t auto close