[Bug] The catch block in Step with 'continueOnError' in try block should be used only in failed Step(s), not in all consecutives successful Steps in Test

[jingav]

chainsaw version Version

v0.2.8

Description

I have a Test with multiple Steps each with try/catch blocks. I added 'continueOnError' to all try blocks with True value to get all errors in one run.

The side effect of this change is that now all (even successful) Steps in that Test print catch blocks. This is polluting report with unnecessary information meant only for troubleshooting in case failure.

Btw. what's the relationship between execution.failFast in .chainsaw.yaml and this 'continueOnError'? It seems that execution.failFast does nothing. Expectation would be that I don't have to specify continueOnError per step and will just control it by failFast in config. And maybe, this continueOnError would overwrite global setting on Step level.

Steps to reproduce

apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
  name: envconfigs
spec:
  steps:
  - name: Assert profile A
    try:
    - continueOnError: true # default is false
      assert:
        file: asserts/envconfig-profile-a.yaml
    catch:
    - describe:
       ....
  - name: Assert profile B
    try:
    - continueOnError: true # default is false
      assert:
        file: asserts/envconfig-profile-b.yaml
    catch:
    - script:
       ....
  - name: Assert profile C
    try:
    - continueOnError: true # default is false
      assert:
        file: asserts/envconfig-profile-c.yaml
    catch:
    - describe:
       ....
    - script:
       ....

Expected behavior

Imagine that assert for profile A is failing. I would expect that only catch from this step will be in a report.

However, there are also catch and script outputs from profile B and C in the report which is not wanted.

Screenshots

No response

Logs

No response

Slack discussion

No response

Troubleshooting

• [X] I have searched other issues in this repository and mine is not recorded.