Discussion about the threshold score of Scorecard

[fredgan]

Hi, folks, Recently, Our company used scorecard to evaluate all the OSSs we used. Yes, we find many OSSs with low score. Since then, we tried to communicate with many OSS communities, and the score of some of them have improved, such as they added the Security.md, added the Branch Protection, and so on.

However, some won't reply us and some declined to us. These projects indeed are in risk. So we need to give up or forbid some projects. I try to make some rules, but there's no reference. Maybe a simple rule is OK: the score must not be under 4.0 (or other threshold value) . Because I don't know any reference score threshold value, 4.0 is too subjective, it's hard to persuade anybody. I'd like to know what's your idea about this from scorecard maintainer's point. Any thinking or suggestion is great appreciated.

[markeverettbeck]

I'm a little curious with this too. My security team leader is anti open source and I'm looking to use this tool to help justify that not everything is scary.

I've started adding issues on their repos hoping that will get some attention, especailly if there is a vulnerability found.

One thing that could help a lot is on the main site for OpenSSF or oven just the Scorecard page make the Report cards searchable. I found this url (https://scorecard.dev/viewer/?uri=github.com//terraform-docs//terraform-docs) by chance in the documentation when trying to set up a manual scan, but it would create a lot more visiblity and awareness if these pages were searchable (especailly in google).

Creating official badges for apps consistantly with a score over 8 for example would encourage use and contribution.

[spencerschrock]

Howdy, we've had some discussion around threshold scores in the past (https://github.com/ossf/scorecard/issues/4219). I recommend reading this section in our README and my comment in the other issue.

I don't know any reference score threshold value, 4.0 is too subjective

I agree, that using any score is subjective. I think looking for individual behaviors is much easier to reason about.

e.g:

• project is not archived
• project has a security.md
• project uses branch protection (there are a few variations here)
• project doesn't have dangerous workflows

However, some won't reply us and some declined to us.

We have run into this in the past when we had some contributors trying to make changes upstream. Ultimately it's up to the maintainer, and not everyone will be receptive. We had success with certain checks more than others. Good merge rate:

• Dangerous-Workflow
• Token-Permissions

Less so:

• Pinned-dependencies
• SAST
• Dependency-Update-Tool

it would create a lot more visiblity and awareness if these pages were searchable (especailly in google)

In terms of google, they are as far as I know (or at least a handful) for pages that link to their scores https://www.google.com/search?q=site%3Ascorecard.dev%2Fviewer

[gcanlin]

@spencerschrock Just a quick side note — do metrics like code review and branch protection really make sense for personal projects? I know that Scorecard mentions in checks.md something like "try to recruit more maintainers to the project who will be willing to review others' work", but many projects are intentionally developed by a single person. While they might accept external PRs, they don't plan to add others as maintainers.

In such cases, do we really need to include metrics that assume multi-maintainer collaboration? For solo-maintained projects, PRs from others naturally require review by the sole maintainer, but tools can't always detect this. And contributors don't have permission to delete branches anyway.

So I’m wondering — should we consider differentiating security evaluation metrics based on the project type (e.g., personal vs. community-driven)? For example, maybe personal projects shouldn’t be evaluated on code review or branch protection. Has OSSF considered this kind of customization in Scorecard? I’d like to refer to any prior thinking or guidance on this.

[spencerschrock]

do metrics like code review and branch protection really make sense for personal projects

I think it depends on the check. Obviously Code-Review is one that is unattainable for single developer projects. I think there's still value in branch protection to protect the dev from making a mistake.

So I’m wondering — should we consider differentiating security evaluation metrics based on the project type (e.g., personal vs. community-driven)?

It's been discussed briefly, for example last year in a pitch I made to treat Scorecard as a linter (with minimal rules enabled by default), I was saying there could be different configurations for different projects. https://docs.google.com/document/d/1I5vtZWa0_64ruFP_MrSaTgzQrAHfh_NknA2feVOHdD4/edit?tab=t.0#heading=h.ktiifbvg64v

[github-actions[bot]]

This issue has been marked stale because it has been open for 60 days with no activity.

[gilescope]

I think you'd get more engagement if there was more flexibility. An installer project or a github action probably isn't the kind of project where fuzzing is appropriate and would just lead to more CO2 in the environment.

For branch protection it's a high risk issue no matter how you score it seems. I.e. if you're one point off passing it's not a low, and two points off passing it's not a medium. Screaming high risk at the slightest infraction means the message looses impact.