Feature: markdown export of Scorecard results

[lucasgonze]

Is your feature request related to a problem? Please describe.

It is difficult to turn scorecard outputs into workflows. The output data is verbose and has to be manually massaged into actionable tasks.

Describe the solution you'd like

The scorecard should be able to output markdown of a checklist to mitigate issues identified by the metrics. The checklist would be designed to paste into a ticket in a tool like Gitlab issues, Github issues, or Jira.

The user experience of the output would be oriented towards actions. Rather than "SAST tool is not run on all commits", "Set SAST tool to be run on all commits."

To my mind the outline-oriented output of the Best Practices badge would be ideal, but I'm not sure it would fit Scorecard's data.

Describe alternatives you've considered

1. Auto-create a kanban. This is less desirable because it varies for every vendor.
2. Generate CSV to import into workflow tools. This is less desirable because there is already JSON and because there is no one format.

Additional context

• https://github.com/ossf/wg-best-practices-os-developers/issues/344
• https://github.com/ossf/tac/issues/169
• Discussed in WG meetings on Jan 8, 2024 and Jan 16, 2024

[evverx]

The scorecard should be able to output markdown of a checklist to mitigate issues identified by the metrics

To be honest it doesn't look particularly convenient to me. In 2024 it should be possible to press a button to create an issue and that issue should be closed automatically when it's resolved. (On GitHub it's kind of partly already possible if the scorecard action is enabled in SARIF mode. Issues can be created by pressing buttons on the security tab)

(Generally before improving anything it would probably make sense to figure out how most projects are managed and cover the most popular use cases. I'm 99% sure that projects using separate products to manage their workflow probably fall into the corporate open source category and they probably should have the engineering resources to parse json, convert it into whatever format they need and pass it on with their internal tokens and so on)

The user experience of the output would be oriented towards actions.

That sounds great. It would be nice to integrate it into scorecard dashboards like https://securityscorecards.dev/viewer/?platform=github.com&org=ossf&repo=scorecard one way or another and make things easily copy-pastable.

[spencerschrock]

Thanks for the issue, and for discussing the maintainer experience. We know it isn't in a good place right now, and are hoping to make it better this year.

In a perfect world, all of the things Scorecard flag will have remediation instructions. This looks like different things for different checks/heuristics. We're in the middle of writing these, and how verbose it is, and when to show it, but we'd like for them to be there. There's still the issue of format as @evverx mentions, and right now that format is parsing JSON.

If you run with an older release (we are temporarily suppressing these as part of the ongoing work we're doing), e.g. v4.10.0 release, you'll see something like this for a very small set of problems Scorecard identifies:

$go run main.go --local=. --format json --show-details | jq
{
  "checks": [
 
    {
      "name": "Pinned-Dependencies",
      "details": [
        "Warn: containerImage not pinned by hash: Dockerfile:16: pin your Docker image by updating golang:1.19 to golang:1.19@sha256:3025bf670b8363ec9f1b4c4f27348e6d9b7fec607c47e401e40df816853e743a",
        ...
      ],
      ...
    },

To my mind the outline-oriented output of the Best Practices badge would be ideal, but I'm not sure it would fit Scorecard's data.

Can you provide a link for reference?

On GitHub it's kind of partly already possible if the scorecard action is enabled in SARIF mode. Issues can be created by pressing buttons on the security tab)

The code scanning dashboard, and PR annotations is one place we'd like to see these SARIF issues auto filed and resolved. Both are in a sad state right now, either due to false positives or being experimental. But is something we're hoping to revisit after the current work.

make things easily copy-pastable.

Again, for most things we don't have this fix auto-generation, but at the very least this desire is built into our code. (I know your position with regard to links to StepSecurity, so trying to not dive into that here).

https://github.com/ossf/scorecard/blob/f1d7a620595d42fa0953da253fed8303c3ed2e1b/finding/probe/probe.go#L43-L47

[lucasgonze]

Can you provide a link for reference?

https://www.bestpractices.dev/en/projects/7029

[evverx]

https://www.bestpractices.dev/en/projects/7029

I don't think that that paragraph (along with the details) is useful in the context of scorecard. It doesn't provide enough details to go and turn on the CodeQL action for example (as far as I can see CodeQL isn't even mentioned there. It points to Wikipedia where it's mentioned but it's still far from any actual actions). It isn't helpful for projects already using SASTs that scorecard can't recognize either. I think https://github.com/ossf/scorecard/blob/49c0eed3a423f00c872b5c3c9f1bbca9e8aae799/docs/checks.md#sast is much more useful in the context of scorecard. That being said it would of course be better if it was easier to arrive at the remediation steps and UIs targeting projects should focus on "remediation steps". (UIs targeting consumers should focus on other things though)

We're in the middle of writing these, and how verbose it is, and when to show it, but we'd like for them to be there.

I think those "motivation", "outcome", "remediation" with "effort" things should help to show different things to different people. Either way it feels like all that has already been discussed elsewhere.

[github-actions[bot]]

This issue has been marked stale because it has been open for 60 days with no activity.

[lucasgonze]

The bot missed that the ticket had been added to the Scorecard - NEW board. We should reopen this.

[github-actions[bot]]

This issue has been marked stale because it has been open for 60 days with no activity.