Improve Score Reporting: Pinned-Dependencies false negative reports

[azeemshaikh38]

In ecosystems like Python and NPM, Pinned-Dependency check can give a score of 10, but their manifest files (requirements.txt and package.json) may actually contain unpinned dependencies. Let's improve our reporting here to show that only certain filetypes are being inspected so that a we don't give a score of 10 to repositories where deps can still be unpinned.

[azeemshaikh38]

/assign to @raghavkaul

[laurentsimon]

The check does not look for requirements.txt and package.json. The check looks for commands that don't treat these files as read-only and / or don't fail if there are unpinned deps in these files.

[raghavkaul]

Today we get dependencies through checks/raw/shell_download_validate_test.go. I think the issue arises if a library doesn't have build steps that point to a deps file (either through lack of build steps or lack of deps file). There's also cases where there may be multiple deps files (e.g. for big repos).

Scorecard could try to scan as many deps files as possible and log its findings, but only compute the scorecard score based on the 'authoritative' deps file that's used for artifact generation, which it can determine by traversing build files, or just finding the file at the repo root when there's only one. Wdyt?

[laurentsimon]

Today we get dependencies through checks/raw/shell_download_validate_test.go. I think the issue arises if a library doesn't have build steps that point to a deps file (either through lack of build steps or lack of deps file). There's also cases where there may be multiple deps files (e.g. for big repos).

I don't follow. If there is no dependency file, it cannot be pinned in most ecosystems. In fact, it's bad practice to pin without a dependency files because renovatebot / dependabot do not parse commands and won't send updates. If the commands used to install dependencies honor the dependency file, then it follows that there is a dependency file used which is pinned.