ossf
Improve Score Reporting: OSV coverage is limited
We only query the OSV database for vulns on a commitSHA so we are extremely limited on the vulns we can report through Scorecard. There is a high possibility that we are rating a commitSHA as 10 on Vulnerabilities check but it actually has reported Vulnerabilities elsewhere. This will quickly erode users trust in Scorecard, need to fix this one ASAP.
I think this just boils down to the issue of not having a good repo -> package level mapping. Most vulnerability databases do not track commit level information, and when they do, it's sparse and incomplete.
Perhaps we should change the meaning of this Vulnerabilities check. Maybe we should instead check if a project has known vulnerabilities in its dependencies. This is much easier to check (because we just scan a project's manifest files) and provides a good signal about a project's hygiene when it comes to dealing with vulns.
We're building a tool in OSV that does exactly that (https://github.com/google/osv.dev/tree/master/tools/osv-scanner#scanning-a-lockfile), which could be integrated here.
@azeemshaikh38 - can you share an example where this false-negative scenario plays out? We have been considering open-sourcing our vulnerability database and we do have significant repo-package and commit-version mapping data that may be able to address the issue.
With https://github.com/google/osv-scanner, we have a way of measuring how many dependencies are affected by known vulnerabilities for most repos. This could be a great way to augment the current vulnerabilities check and make it more useful.
CC @another-rex who owns osv-scanner.
This is really nice, thanks for sharing. How would you envisage scorecard ingesting the results? Are you planning on exposing a Go API? For example, wrapping the CLI's options into an API (like we did for Scorecard https://github.com/ossf/scorecard/blob/main/main.go)?