Show controls that failed

[spencer-cdw]

I am using saf as part of a pipeline workflow where saf will cause the image build to fail if it exceeds a threshold. This works well.

If a control isn't passing, we want to be able to quickly view the control id as part of the pipeline, so we can easily address the failures

Feature request:

Add a command line flag that prints out control status to STDOUT/STDERR so that it can be consumed as part of a pipeline

Workaround

The closest workaround I've found is by misusing the saf generate threshold using -c option, combined with cat and yq

saf generate threshold -i inspec.json -c --output /tmp/foobar; cat /tmp/foobar | yq .error

critical:
  max: 4
  controls:
    - cis-dil-benchmark-5.2.17
    - cis-dil-benchmark-5.3.1
    - cis-dil-benchmark-5.4.1.5
    - cis-dil-benchmark-6.2.13
high:
  max: 0
medium:
  max: 0
low:
  max: 0
total:
  max: 4

Solution ideas

1. Port the -c logic from saf generate threshold to saf summary

   saf summary -i inspec.json -c

Additional thoughts

It would be nice if I could use something like --filter directly from saf, and not need to install yq on my builders

e.g. `saf summary -i inspec.json

[aaronlippold]

well we already add jq, ca-certificate, etc on the containers we could add yq as well and look into adding a --filter flag.

In another issue, can out outline how you would like --filter to work and across which sub-commands

[spencer-cdw]

These 2 PRs combined would create a good workaround:

https://github.com/mitre/saf/pull/1249 https://github.com/mitre/saf/pull/1250

[aaronlippold]

This brings up an interesting question around Details Level much like the idea of Log Level

It would seem to me that 3-5 levels of verbosity or depth would likely fall out of this:

1. current default of general bucket totals with breakdown of HML
2. proposed control level detail from the bucket totals
3. totals only in Pass, Fail, NA, NR, Error - no HML
4. ... others?

[spencer-cdw]

Current work around

Use save generate threshold -c and save to a file Use a python script to convert the threshold file from yaml to json use jq with jq 'try .error.critical.controls catch empty' <<< $inspec_report_json to look for any errors. If there are any controls that are failing that don't have a waiver, then fail the pipeline.

Basically if the threshold looks like this, then it is a pass

critical:
  max: 4
  controls: #notice how this is empty? 
high:
  max: 0
medium:
  max: 0
low:
  max: 0
total:
  max: 4

But if controls are listed, then fail and print out what those failed controls are

critical:
  max: 4
  controls:
    - cis-dil-benchmark-5.2.17
    - cis-dil-benchmark-5.3.1
    - cis-dil-benchmark-5.4.1.5
    - cis-dil-benchmark-6.2.13
high:
  max: 0
medium:
  max: 0
low:
  max: 0
total:
  max: 4

      - name: Check inspec
        shell: bash
        run: |
          saf view summary -i ${{ env.INSPEC_REPORT }} 2>/dev/null
          saf generate threshold -i ${{ env.INSPEC_REPORT }} -c --output ${{ env.PKR_VAR_root_file_path }}/output/${{ env.CLOUD }}/${{ matrix.image }}.yml 2>/dev/null
          cat ${{ env.PKR_VAR_root_file_path }}/output/${{ env.CLOUD }}/${{ matrix.image }}.yml
          cd actions/yaml2json
          pip3 install -q -r requirements.txt
          inspec_report_json=$(python3 yaml2json.py ${{ env.PKR_VAR_root_file_path}}/output/${{ env.CLOUD }}/${{ matrix.image }}.yml)
          error=$(jq 'try .error.critical.controls catch empty' <<< "$inspec_report_json")
          if [[ $error == "null" ]]; then
            echo "Inspec Passed" >&2
          else
            echo "Inspec Failed" >&2
            echo "$error"
            exit 1
          fi
        if: success() || steps.inspec_json_report.outcome == 'success'