BUG SAST tool check runs on doc-only commits.

[ianlewis]

The slsa-github-generator repo currently omits running the CodeQL action on commits that are documentation or yaml only as the CodeQL action is slow doesn't support markdown or yaml anyway.

However, scorecard dings us for not running CodeQL on all of our commits. It would be nice if the scorecard could detect if the commits were documentation-only changes or not.

[azeemshaikh38]

Thanks @ianlewis. IIUC, there are static analysis tools for config/workflow files. So checking for SAST tools on commits which only change YAML would be a valid usecase?

If we were to limit ourselves to doc files (like .md) only, my concern is:

• retrieving the changed files through GitHub API might significantly increase API token usage
• would not work for projects which use Gerrit or other such code review tools
• and is checking for commits which only modify .md file a strong enough usecase to merit investing into?

[ianlewis]

Thanks @ianlewis. IIUC, there are static analysis tools for config/workflow files. So checking for SAST tools on commits which only change YAML would be a valid usecase?

I think that's a fair point. It would be nice if scorecard had support for them.

If we were to limit ourselves to doc files (like .md) only, my concern is:

• retrieving the changed files through GitHub API might significantly increase API token usage
• would not work for projects which use Gerrit or other such code review tools
• and is checking for commits which only modify .md file a strong enough usecase to merit investing into?

Maybe the solution is expanding the number of SAST tools that scoreccard checks for?

Scorecard itself has SAST like behavior for its permissions checks on GitHub workflows. I know several projects that have found those valuable.

[azeemshaikh38]

@laurentsimon do you know if we consider scorecard-action itself as a SAST tool?

[laurentsimon]

We currently do not, but we should revive https://github.com/ossf/scorecard/pull/1487 and improve the SAST check to include it. The PR is too API intensive, and we need to solve this problem before merging it.

[AlexGustafsson]

I'd like to "revive" this issue. By default, the CodeQL actions activated through GitHub Advanced Security run on all commits, for all configured languages. Which work well with the Scorecard action. But as the CodeQL checks are slow, it can be costly to run on examples like OP's - where only documentation is changed.

To add configuration to the CodeQL actions, GitHub documents "advanced configuration" for CodeQL, with the first example showing how to only run the applicable checks for the changed files: https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/customizing-your-advanced-setup-for-code-scanning#avoiding-unnecessary-scans-of-pull-requests.

Is there any recent development (last ~two years) that would warrant an updated take on this issue?

[spencerschrock]

There should be pretty good partial credit in that case, as we give 70% weight for having a workflow call codeQL, and 30% for how many of the PRs are checked. Assuming doc-only commits aren't super frequent, this isn't as big of a problem.

Is there any recent development (last ~two years) that would warrant an updated take on this issue?

Adam has been adding support for more SAST tools too, as mentioned in an earlier comment here. Such as infrastructure as code ones in #4687

[AlexGustafsson]

Thanks for the update! Yes, the weighing seems to be fair and helps a lot with the documentation use case.