CLI command to evaluate policies

[jiparis]

it would be useful to have something like chainloop policy eval that receives a material and a policy (file, URL) and returns a report with potential violations. I would be used in manual control gates

[migmartri]

I'd love if this command has a 'development mode` (flag) that allows us to see the input, data,linting errors, and evaluation output, similar to what the rego playground does.

[jiparis]

I'd love if this command has a 'development mode` (flag) that allows us to see the input, data,linting errors, and evaluation output, similar to what the rego playground does.

I think opa eval might be of help for this:

> cat input.json
{"components": [{"name": "thename", "bom-ref": "theref", "licenses": []}]}

> opa eval -d cyclonedx-licenses.rego -i input.json 'data.cyclonedx_licenses.violations'
{
  "result": [
    {
      "expressions": [
        {
          "value": [
            "Missing licenses for thename (theref)"
          ],
          "text": "data.cyclonedx_licenses.violations",
          "location": {
            "row": 1,
            "col": 1
          }
        }
      ]
    }
  ]
}

[migmartri]

@jiparis I just wrote a quick spec, what do you think?

Overview

Today, to write a custom policy, you need to

• Copy our rego template
• Create a policy.yaml file
• Use rego-playground to develop the policy code by providing some test data
• use rego linter
• run it during an attestation process to see if it works

This approach has some issues:

• Our policies are more than just rego scripts, their root is a policy.yaml file that's in charge of defining arguments, encode different material type paths, etc. This is not easily tested using the process above.
• It's not trivial to know in advance if a writen rego script meets our content standards (result struct, skip_reason, etc) defined in our template.
• A test run is an expensive process. It requires attaching the policy to a contract, running an attestation, or manually crafting some test data in rego-playground.

Goals

To have a development environment to write, lint, and test custom chainloop policies.

chainloop policy develop [subcommand]

init [directory]

Bootstraps a chainloop policy and example script based on our template. This template should be workable out of the box

lint

By default it load a policy.yaml file in the current directory and perform

• rego script(s)

  • linting (maybe regal)
  • opa fmt check
  • chainloop result structure check

• chainloop.yaml

  • marshall content / proto validate

eval --material [material-file] --kind [...] --annotations ...

• Make sure there is a path in the policy.yaml for that kind
• Perform a full evaluation of the policy against the provided material type, provide any annotations
• Should work for the provided materials or attestations

[jiparis]

For policy evaluations, we have something like this internally that can be used as inspiration. It essentially takes a policy file (the yaml spec), a material (the raw content, like an SBOM, report, or even an ATTESTATION), and the policy arguments:

func runTest(policyFile, materialFile, kind string, inputs map[string]string) error {
	schema := &v1.CraftingSchema{
		Policies: &v1.Policies{
			Materials: []*v1.PolicyAttachment{
				{
					Policy: &v1.PolicyAttachment_Ref{Ref: fmt.Sprintf("file://%s", policyFile)},
					With:   inputs,
				},
			},
			Attestation: nil,
		},
		PolicyGroups: nil,
	}

	material := &v2.Attestation_Material{
		M:            &v2.Attestation_Material_Artifact_{Artifact: &v2.Attestation_Material_Artifact{Id: "test-material"}},
		MaterialType: v1.CraftingSchema_Material_MaterialType(v1.CraftingSchema_Material_MaterialType_value[kind]),
	}

	logger := zerolog.New(os.Stderr).Level(zerolog.WarnLevel)
	v := policies.NewPolicyVerifier(schema, nil, &logger)
	evs, err := v.VerifyMaterial(context.Background(), material, materialFile)
	if err != nil {
		return err
	}
	return nil
}

[migmartri]

btw, what I mentioned today about making sure we are running the policies with the right capabilities, I think this is done automatically already if you eval the policies with our engine https://github.com/chainloop-dev/chainloop/blob/17dbde7fbf97298c025e935691dd2de7c23c167c/pkg/policies/engine/rego/rego.go#L233