Capture and Process Diagnostic and Metric Summary Output in codeql-action/analyze@v2

[swakhil-strobes]

I am currently utilizing the github/codeql-action/analyze@v2 action to execute CodeQL analysis within my GitHub Actions workflow. While the action successfully generates diagnostic and metric summary output, I am facing challenges in capturing and processing this output to make decisions regarding the success or failure of the workflow based on critical or warning data.

I have attempted various approaches, such as printing the returned output for debugging purposes and inspecting the log files, but unfortunately, I have not been able to locate any stored logs or find a reliable method to capture the necessary information.

one of the logic I tried

- name: Perform CodeQL Analysis
  id: codeql
  uses: github/codeql-action/analyze@v2
  with:
    category: "/language:${{matrix.language}}"

- name: Print Output
  run: |
    echo "CodeQL Analysis Output: ${{ steps.codeql.outputs }}"
    echo "${{ steps.codeql.outputs }}"
    echo "${{ steps.codeql.outputs['db-locations'] }}"
    echo "${{ steps.codeql.outputs['sarif-id'] }}"

I would greatly appreciate any guidance or suggestions.

Thanks!

[aeisenberg]

So, you are looking to fail the codeql analysis workflow if there are any alerts that are discovered? This is not the typical way that the workflow is run. Usually, after analysis, all alerts are sent up to the code scanning back end so that duplicated alerts, false positive alerts, ignored alerts, etc can be ignored. There is another check run that will fail if codeql finds any true positive alerts.

It sounds like you are trying to do something different. Can you explain what your goals are?

[swakhil-strobes]

Let me explain more clearly. My codeql analysis produces below summary diagnostic & metric data

Analysis produced the following diagnostic data:

|                    Diagnostic                     |       Summary        |
+---------------------------------------------------+----------------------+
| Python extraction warnings                        | 1 result (1 warning) |
| Could not process some files due to syntax errors | 1 result (1 warning) |
| Successfully extracted Python files               | 60 results           |


Analysis produced the following metric data:

|                         Metric                          | Value  |
+---------------------------------------------------------+--------+
| Total lines of user written Python code in the database |   1477 |
| Total lines of Python code in the database              | 126978 |

Now, my objective is to fail the workflow based on the generated summary data. I have attempted to implement an additional step that captures the output and applies regular expressions to identify any critical or warning messages within the diagnostic or metric data which is not working. In the provided example analysis, I would identify warnings in the "Python extraction warnings" and "Could not process some files due to syntax errors" sections, resulting in the workflow being failed.

The step I had added which didn't work:

   - name: Perform CodeQL Analysis
      id: codeql
      uses: github/codeql-action/analyze@v2
      with:
        category: "/language:${{matrix.language}}"

   - name: Check for critical or warnings
      run: |
      # Retrieve the number of critical issues from the CodeQL analysis results
        critical_issues=$(jq '.annotations[] | select(.properties.severity == "critical")' "${{ steps.codeql.outputs.codeql-results }}")
    
      # Fail the build if there are critical issues
        if [[ $critical_issues ]]; then
          echo "Critical issues detected:"
          echo "$critical_issues"
          exit 1
        fi

Additionally, I would like to emphasize that despite my efforts, I have been unable to find a suitable solution to capture and process the diagnostic and metric summary data in order to determine whether the workflow should fail or not based on the presence of critical or warning messages. Therefore, I am seeking guidance and assistance from the team to help me achieve this desired outcome.

[aeisenberg]

Thanks for the explanation. Just to be sure I'm understanding, your goal is to fail the workflow if there are any extractor warnings?

I will ask the people involved with diagnostic information on the best way forward.

[aeisenberg]

In the analyze action, set the output parameter as an input in the with block. This will determine the directory where the SARIF is saved. Once you have that, in the next step, use jq to look for toolExecutionNotifications with level: warning or level: error.

Be aware, though, that diagnostic information is generally meant to be internal. Sometimes there will be warnings or messages around things that are unrelated to the code being extracted.

In general, messages like Could not process some files due to syntax errors are things that are better tracked by using a source code linter. Perhaps a more robust way of handling this situation is to run a python linter like ruff as part of CI.