Add configurations for CI to fail on OSH scan failures and new findings

[siteshwar]

This is a follow up on https://github.com/packit/packit/discussions/2371#discussioncomment-10474198

We should add two separate configuration options to cause CI to fail on scan failures and new findings:

• fail_ci_on_scan_failure should cause CI to become red if OSH scan fails.
• fail_ci_on_new_findings should cause CI to become red on new findings.

Both of these options should be kept false by default. Because there may be issues with buildroot that can cause a scan to fail, or there may be large amount of false positives for certain projects.

[lachmanfrantisek]

Thanks @siteshwar for writing this down.

As a first thing, we need to resolve the reporting in general: https://github.com/packit/packit-service/issues/2516

[mfocko]

I would probably prefer blocking attribute… On GitHub we could set non-blocking to neutral status, if it fails (that doesn't block merging)

[siteshwar]

• fail_ci_on_new_findings should cause CI to become red on new findings.

On a second thought, the status should not be "fail", it should be "action_required" on new findings. Also, it should be "neutral" if there is a new finding, but the CI is not configured to fail.

[siteshwar]

This may be more complicated then it looked initially, as we plan to upload SARIF to CodeQL and it has its own checks for severity of the findings that determines the status of the CI.

[mfocko]

Can't the CodeQL replace the checks? :thinking:

[siteshwar]

Can't the CodeQL replace the checks? 🤔

It seems configurable, but the default setting hides results from the user.

We can only keep the osh-diff-scan check and avoid uploading to CodeQL. Check should directly reference the final html report from OpenScanHub.

[siteshwar]

On a second thought, the status should not be "fail", it should be "action_required" on new findings. Also, it should be "neutral" if there is a new finding, but the CI is not configured to fail.

It should be probably renamed to "action_required_on_new_findings".

[lachmanfrantisek]

A note from today's meeting, we might want to consider having this for all the jobs.