Output Scorecard results as in-toto attestation

[adityasaky]

In addition to existing formats, it'd be handy to have scorecards result emitted as an in-toto attestation. The attestation framework is described here: https://github.com/in-toto/attestation.

Why?

in-toto's attestation framework is a structured way to emit software supply chain specific claims. These claims can then be consumed by a policy engine like an in-toto "layout". Having scorecard results as an attestation would allow for its use alongside other attestations like SLSA provenance, test results, SCAI, as well as some popular SBOM formats like SPDX and CDX.

What would it look like?

The document would have a lightweight in-toto Statement and the scorecard results themselves would be within it as a predicate. Each predicate is identified using a type which is a URI. I think using the URL for this repository + version would work, but I'll leave it to the scorecard maintainers to decide!

Open Questions

in-toto attestations are typically cryptographically signed and embedded in a signature envelope like DSSE. I'm not yet sure if scorecard supports that or intends to. By enabling that, the policy engine can also assert that the results were generated by a trusted entity (such as the correct CI workflow). Curious to hear your thoughts!

[idunbarh]

I think its a great idea. There already is some existing work in this area for binary authorization attestation and expanding it to include in-toto makes sense.

• https://github.com/ossf/scorecard/tree/main/attestor

[github-actions[bot]]

This issue is stale because it has been open for 60 days with no activity.

[adityasaky]

Whoops!

I'll spread the word a bit in the in-toto community to see if someone's interested in making this happen.

[github-actions[bot]]

This issue is stale because it has been open for 60 days with no activity.

[bobcallaway]

@spencerschrock @laurentsimon @kpk47 curious as to your thoughts on this

[spencerschrock]

I agree with Ian that we do a similar thing already with the scorecard attestor code, although it does slightly different things (cc @raghavkaul). We also do something similar already with ossf/scorecard-action, where we use hashrekord log entries to accept crowd-sourced scores for the API.

action side: https://github.com/ossf/scorecard-action/blob/c64aac8aa26f9701232d21f7382097ea7044cd8d/signing/signing.go#L81-L82 webapp side: https://github.com/ossf/scorecard-webapp/blob/28b861a9b9712b3a097ecf94e1057f3a7e802668/app/server/post_results.go#L133-L148

[raghavkaul]

Today the attestor subsystem attests whether a repo passes a YAML-based policy, but maybe if scorecard signed/attested to the entire result output, that would give the score consumer more information about a repo.

[github-actions[bot]]

This issue has been marked stale because it has been open for 60 days with no activity.

[marcelamelara]

Pinging @zachariahcox, @TomHennen and @meder on this re: potential connection to the SLSA source/dependency track.

[meder]

Thanks, @marcelamelara.

One question I had was whether each scorecard check should have its own predicate type to enable composability where different tools can generate the same type of attestations and consumer is able to distinguish between different attesters.

Has this been discussed somewhere?

[adityasaky]

I don't know if it's been discussed elsewhere, but I think as a starting point a single predicate type might suffice. I'm unaware of the different ways scorecard is deployed: are there cases where a subset of checks are used?

[spencerschrock]

I don't know if it's been discussed elsewhere, but I think as a starting point a single predicate type might suffice. I'm unaware of the different ways scorecard is deployed: are there cases where a subset of checks are used?

The Scorecard binary supports all checks, but depending on who's running it the results may cover a different subset. Our API currently can return either 19 or 16 checks due to limitations we have running the analyses at scale. Additionally we have another result format now, so would this involve at least 4 predicate types?

1. The full Scorecard output (JSON)
2. Output for a single check (JSON)
3. full Scorecard output, but with "probes" (discrete heuristics which make up the check) instead of checks (JSON, but different structure)
4. Output of a single probe (see 3)

generic Scorecard container (1. or 3. depending on check/probe)

"predicateType": "https://scorecard.dev/Result"
"predicate": {
    "checks": [
    {
        "name": "Code-Review",
        "score": 10,
    }
    {
        "name": "Some-Other-Check",
        "score": 7,
    }
    // ....
    ],
}

a single check/probe, generic:

"predicateType": "https://scorecard.dev/CheckResult"
"predicate": {
    "name": "Code-Review"
    "score": 10,
   // or some other check details
}

a single check/probe specific:

"predicateType": "https://scorecard.dev/CodeReviewResult"
"predicate": {
    "score": 10,
   // or some other Code-Review specific details
}

[adityasaky]

Thanks for the clarity, @spencerschrock! I think we could do something like 1 to start, maybe even closer to:

"predicateType": "https://scorecard.dev/result/v0.1"
"predicate": {
    "codeReview": 10,
    "someOtherCheck": 7,
    ...
}

Since predicate types can be versioned, we can control for the checks included and don't have to leave it entirely open ended. We could also separate out the types for the 19 and 16 cases, maybe, so the consumer has no confusion about which set of checks were run.

Once we have more clarity, we can evaluate per-check predicates. However, unless we expect to have 100s of checks and a massive attestation, I wonder if it's fine for a consumer to get a full attestation even if they care about the results of only one check.

Thoughts?

[ramonpetgrave64]

Another great thing to have is the ability to attest to scorecard results over a time period, or to verify that certain checks were not violated in previous x months or x releases. It seems like this could be possible for the Scorecard App with webhooks for Github Apps.

• https://docs.github.com/en/webhooks/webhook-events-and-payloads#repository_ruleset

[TomHennen]

Hello Scorecards!

Over in SLSA we've been thinking about how Scorecards might be able to attest to the SLSA Source Track (draft). Specifically we're thinking it might be easy enough for Scorecards to issue a verification summary attestation for GitHub repos it monitors. Given the current requriements I think it might be quite easy for Scorecards to issue a VSA attesting to SLSA Source Level 2 by simply indicating if branch protection is enabled.

What do you think? Is this the right place to have this discussion? Would you prefer a separate issue?

Thanks!

[spencerschrock]

I think it might be quite easy for Scorecards to issue a VSA attesting to SLSA Source Level 2

I worry about Scorecard needing to produce different intoto predicates types to support some arbitrary number of external requirements that consumers may want. Which is partly why I suggested a Scorecard specific predicate in https://github.com/ossf/scorecard/issues/3352#issuecomment-2359417591 stating we observed some behavior(s). Consumers who want to build policy on top of that can consume our VSA and make a derivative VSA. (Is this a thing?)

[github-actions[bot]]

This issue has been marked stale because it has been open for 60 days with no activity.

[puerco]

OK, here is a PR with a change to add --format=statement and a predicate suggestion based on the current JSON output:

https://github.com/ossf/scorecard/pull/4491