Feature: Scorecard policy that helps evaluate results for GCP Binary Authorization

[raghavkaul]

trafficstars

Current Behavior

Today, scorecard's policies can be used to enable/disable certain checks. As per #1405 , the --policy flag was disabled in favor of implementing a settings-based policy file.

Desired Behavior

One motivation for scorecard policies is to use them to make decisions about the runtime authorization (e.g. Google Cloud Binary Authorization) for an application. For example, an operator may want to use scorecard to implement the rule Don't run this application unless the SCM repo used to build this application is free from Binary Artifacts AND unless the SCM repo mandated two-party review.

We can think of 'policies' as a way of using all of scorecard's data to answer questions an operator may have about an application, as opposed to what is done today, which uses policies to change how scorecard behaves.

One possible approach would take heavy inspiration from the way Allstar policies are written.

This is linked to #1245.

[raghavkaul]

I want to lay out a few decision points for the design of this feature.

Decision Point 1: How detailed should policies be?

There are two approaches to policies. First, Scorecard could expose specific, high-level Policy Verbs or statements. The user could parameterize policy with ints, booleans, or lists of strings, or leave out the verb to ignore it. Scorecard would produce an attestation if all policy verbs are satisfied. This approach is taken by allstar and voucher.

Another more flexible approach to specifying policies, is to allow users to specify arbitrary code or logic over Scorecard data as a Predicate Function. If the user’s policy predicate is satisfied, an attestation would be produced. This approach is taken by cosign.

Decision Point 2: How should policies be expressed?

Policy: None/Hardcoded

To enable this, Scorecard would expose a series of ‘verbs’, similar to the YAML-based policies (e.g. requireApproval if all commits were approved in a PR process for an attestation to be produced).

Verbs would be exposed through CLI options, similar to grafeas/voucher. If all verbs are satisfied, an attestation is produced.

Policy: YAML

In this implementation, users would pass a YAML config with policy ‘verbs’. Each verb reflects a requirement that the SCM repo must pass before being authorized to run. The YAML policy would draw data from raw results. If all conditions are satisfied, an attestation is produced.

Today, Scorecard already supports policy - but this policy enables or disables a Scorecard check entirely, and sets a numeric score cutoff.go/scorecard-binauthz-attestations

Existing Scorecard Policy

type checkPolicy struct {
        Mode  string   `yaml:"mode"`
        Score int      `yaml:"score"`
}

type scorecardPolicy struct {
        Policies map[string]checkPolicy  `yaml:"policies"`
        Version  int                     `yaml:"version"`
}

Proposed Scorecard Policy

type ScorecardPolicy struct {
        // ProtectedBranches : a list of repos that have branch protection enabled
        // All branches in this list must require branch protection
        ProtectedBranches []string `yaml:"protectedBranches"`

        // RequireApproval : set to true to require approval on all PRs, default true.
        RequireApproval bool `yaml:"requireApproval"`

        // ApprovalCount is the number of required PR approvals, default 1. This repo
        // must require at least this many approvals, or the policy check will fail
        ApprovalCount int `yaml:"approvalCount"`

        // DismissStale : set to true to require that this project's SCM repo requires
        //  PR approvals be dismissed when a PR is updated, default true.
        DismissStale bool `yaml:"dismissStale"`

        // BlockForce : set to true to require that this project's SCM repo blocks 
       // force pushes, default true.
        BlockForce bool `yaml:"blockForce"`

        // RequireUpToDateBranch : set to true to require that this project's SCM 
        // repo requires branches must be up to date before merging. Only 
        // used if RequireStatusChecks is set. Default true.
        RequireUpToDateBranch bool `yaml:"requireUpToDateBranch"`

        // RequireStatusChecks set to true to require that this project's SCM repo
        // requires status checks in order to merge into the protected branch. Each 
        // entry must specify the context, and optionally an appID.
        RequireStatusChecks bool `yaml:"requireStatusChecks"`

        // EnforceOnAdmins : set to true to true to require that this project's SCM repo 
        // enforces branch protection rules on administrators as well.
        EnforceOnAdmins bool `yaml:"enforceOnAdmins"`

        // RequireSignedCommits : set to true to require signed commits on protected 
        // branches, default false
        RequireSignedCommits bool `yaml:"requireSignedCommits"`

        // PreventBinaryArtifacts : set to true to to require that this project's SCM repo           // is free of binary artifacts
        PreventBinaryArtifacts bool `yaml:"preventBinaryArtifacts"`

        // ... More TBD!
}

Policy: OPA/Rego

OPA is a policy language used by Kubernetes admission controllers and Envoy. It can be used to specify policy on structured data. Users could use OPA’s top-level allow rule to define the policy, and compose it with any number of sub-rules (See: Appendix: OPA Policy Composability)

To enable this, Scorecard would have to 1) make structured raw data (with a consistent JSON schema) accessible to user-defined OPA rules, and 2) invoke OPA to execute those user-defined rules. As opposed to the YAML and Hardcoded approaches, users could define their own policy verbs.

To make it easier to write OPA rules, we could include simple rules with Scorecard, for user-written rules to import. For example, we could write a Rego rule for “an image must be built with no binary artifacts” called binary_artifacts, which could be used to compose a higher-level policy.

Decision Point 3: Subcommand vs. separate module?

Should the Scorecard CLI have a subcommand that consumes policy and produces an attestation in Google Cloud Container Analysis? Or should it simply produce results in a way that another program can interact with Google Cloud?

Option 1 is that scorecard prints results to file/stdout, another program reads them and creates the attestation. Option 2 is that Scorecard evaluates the policy and creates an attestation if the policy check passes.

[david-a-wheeler]

Thanks! I wonder - the yaml-based approach may have some overlap with the proposed "SECURITY-INSIGHTS" document, per https://github.com/ossf/security-insights-spec.

[raghavkaul]

@david-a-wheeler Thanks for bringing that project to my attention! My initial thought is that the threat model of Security-Insights is different from that of a Binary Authorization policy (or other deployment policy). Whereas Security-Insights is the set of insights about a specific repo declared by the repo owner, the Binary Authorization policy would be a set of standards for any repo declared by a cloud deployment gatekeeper. If repo owners controlled the Binary Authorization/deployment policy, then they could upload a policy that would always pass and always allow deployment, even if the Scorecard score was low and the practices followed were poor.

However, if we were to use this schema on the deployer-side, an allstar-style policy would be easier to use because it would express security requirements more directly. For example, if I wanted to express "The code in an artifact must require 2-party review in its SCM in order to be deployed," in an allstar-style policy format, this would be a one-line change (ApprovalCount: 2), and the implementation of the policy logic would fall on Scorecard maintainers. Using Security-Insights means Scorecard can be more generic and not maintain as much custom logic for each check, but I think expressing the same requirement as a YAML (even if it was sparse, and we did something like semantic YAML matching) would be a higher bar for the end-users of this feature.

[azeemshaikh38]

@laurentsimon @naveensrinivasan @justaugustus - Raghav is ready to start work on this. Please give a shout if there are any concerns here.

[laurentsimon]

This work works well on branch protection because each result is "named", but not well for other checks like token-permissions or pinning. What's the plan for the others?

[azeemshaikh38]

IMO some checks like Branch-Protection, Token-Permissions shouldn't be exposed at all through this policy. I don't see these checks being useful to gate an image built at commit X. The list of checks to expose here IMO are: Bin-Artifact, CI-Tests, Code-Review, Pinned-Deps, SAST and Vulnerabilities (atleast for the initial release).

And for checks that require an ignoreList (Pinned-Deps, Bin-Artifacts) how about something like:

ScorecardPolicy struct  {
  // In YAML, specify:
  // allowedUnpinnedDependencies: nil // Disable this check.
  // allowedUnpinnedDependencies: []  // Enable this check and apply on all.
  // allowedUnpinnedDependencies: // Enable and apply on all except provided exceptions.
  //   - dependency:
  //        filepath: <>
  AllowedUnpinnedDependencies []Dependency{
     Filepath string
     PackageName string
     Version string
  }
  // 
  AllowedBinaryArtifacts []FilePath
}

Wdyt?

[laurentsimon]

IMO some checks like Branch-Protection, Token-Permissions shouldn't be exposed at all through this policy.

do you mean Dependency-Pinning or Branch-Protection? (I'm asking because @raghavkaul 's initial description is heavy on the Branch-Protection check.) I understand that the check is not bound to a commit / source code, though. If this is better suited for AllStar, does this mean we'll ask users to install both? Or will AllStar encompass all the policies we declare here?

I don't see these checks being useful to gate an image built at commit X. The list of checks to expose here IMO are: Bin-Artifact, CI-Tests, Code-Review, Pinned-Deps, SAST and Vulnerabilities (atleast for the initial release).

It's not clear why, for example, Token-Permissions could not be exposed. What's the reasoning?

And for checks that require an ignoreList (Pinned-Deps, Bin-Artifacts) how about something like:

ScorecardPolicy struct  {
  // In YAML, specify:
  // allowedUnpinnedDependencies: nil // Disable this check.
  // allowedUnpinnedDependencies: []  // Enable this check and apply on all.
  // allowedUnpinnedDependencies: // Enable and apply on all except provided exceptions.
  //   - dependency:
  //        filepath: <>
  AllowedUnpinnedDependencies []Dependency{
     Filepath string
     PackageName string
     Version string
  }
  // 
  AllowedBinaryArtifacts []FilePath
}

Wdyt?

[azeemshaikh38]

If this is better suited for AllStar, does this mean we'll ask users to install both? Or will AllStar encompass all the policies we declare here?

Scope here does not include defining a generic policy. This policy definition will only be used for expressing a BinAuthz policy.

It's not clear why, for example, Token-Permissions could not be exposed. What's the reasoning?

It could be exposed, I mean they don't need to be because there doesn't seem to be a strong usecase for a BinAuthz client to gate on Token-Permissions.

[raghavkaul]

The main reason to keep Branch-protection out of the checks is that it's point in time, so it makes sense for Allstar because Allstar checks it during merge-time, but for a deployment-time check, Binary authorization could have been enabled post-hoc.

Also agree with the first set of checks implemented into policy to be:

Bin-Artifact, CI-Tests, Code-Review, Pinned-Deps, SAST and Vulnerabilities

This work works well on branch protection because each result is "named", but not well for other checks like token-permissions or pinning. What's the plan for the others?

It's case by case - everything we express in YAML may have a different check format depending on what makes sense for a deployer to gate on