Clarifying values of `verifiedLevels` in VSA v1

[AdamZWu]

The current documentation seems to refer to two separate ladders.

• The SLSA build ladder, which currently goes from 0 to 3, and the value is dependent on the builder, not policy verification.
• The "FAILED" value is reserved for denoting policy verification failure, which is separate from the SLSA build ladder, but there are no other values mentioned.

Would like clarification, esp. on the "FAILED" value and its use case.

For example, if we have an artifact that is SLSA_BUILD_2, it got checked against a policy which say "must be SLSA_BUILD_3 or above", will the verifiedLevels look like [ "FAILED", "SLSA_BUILD_2" ]?

[joshuagl]

The text describing SlsaResult states that SlsaResult conveys "The result of evaluating an artifact (or set of artifacts) against SLSA."

That is, the VSA describes the SLSA level verified by the VSA's signer, not the level claimed by the artefact's producer.

Further, the verifiedLevels field:

Indicates the highest level of each track verified for the artifact (and not its dependencies), or “FAILED” if policy verification failed.

Users MUST NOT include more than one level per SLSA track.

So, in the example given verifiedLevels = [ "FAILED" ]

cc @TomHennen @kpk47

[AdamZWu]

There are two subtle issues with the interpretation @joshuagl:

1. The "FAILED" is over-broad. Current spec only defined a single track (i.e. SLSA BUILD), so it looks like it could work; however, as soon as we add another track, "FAILED" becomes ambiguous -- which track did it fail? or all of them?

Also, "FAILED" doesn't have the "SLSA_" reserved prefix, so it is in a strange position -- it sort of belongs to the SLSA track (or multiple SLSA tracks?) as it is a possible SLSA verification result, OTOH it also sort of belongs to the non-SLSA tracks, if the verifier also evaluates customer ladders.

2. The SLSA BUILD track doesn't seem to need "FAILED" at all. Since the build level 0 is effectively catching everything, technically nothing can fail that level.

Additionally, I think it would be useful to separately convey "what is the build level of the artifact", and "what is the overall policy verification result". So even if an artifact failed the overall policy verification, knowing that it is built in compliance with SLSA BUILD 2 is better than know nothing.

---

One more thing, it seems the "FAILED" level is serving redundant purpose to the verificationResult: If policy verification failed, verificationResult must already be set to "FAILED", having another "FAILED" in verifiedLevels doesn't provide more information.

[MarkLodato]

Related: #964 by @kpk47

I agree with @AdamZWu about removing FAILED from verifiedLevels.

I also agree it might be a good idea to separate "what did the policy require" from "what did I observe". The set difference would show you why the thing failed. And while we're at it, it might simplify things to no longer have SLSA_BUILD_L<n> imply SLSA_BUILD_L<n-1>. That way it's purely a set. Or alternatively represent it as a map instead of a set of enums.

I wonder if we should have a generic field allowing the verifier to pass along arbitrary bits of information from the input attestations. We have a need for this internally within Google, and I imagine others will as well. I'm not sure how best to do this. Could allow arbitrary JSON, but then it gets complicated and you need to talk about schema. A key/value map might be a compromise.

[TomHennen]

My only concern about including arbitrary bits to be passed along is that, over a large supply chain, the VSA could go very very large. E.g. if the VSA of each dependency included custom details, then the VSA at the end of the chain that needs to summarize all of that could become quite large.

An advantage of a small VSA is that some metadata storage systems have size limitations, and also that they can ideally be evaluated quickly.

So, not that we shouldn't support it, just that we'd need to be careful about guidance about how to include this information, if it gets propagated recursively or not, and guidance about size of the data.

[AdamZWu]

Agree with Tom. The purpose of VSA is to summarize verification, and keeping it lean and mean is essential for facilitating more complex operations to perform well at scale, such as transitive dependency checks.

So some guidance or even requirements such as "no information propagates from dependency VSA without being summarized" and/or "no nesting >2 levels" feels important to me.

[joshuagl]

Thanks for the detailed discussion, I agree with the proposal to remove FAILED from verifiedLevels and to not have SLSA_BUILD_LEVEL_N imply SLSA_BUILD_LEVEL_N-1.

@AdamZWu would you be open to creating a PR?

[AdamZWu]

Sure, will do. Should I target the change to v1.1?

[joshuagl]

Targeting v1.1 makes sense, thanks. We need to be careful how we introduce this so that we don't break our semver. As we support custom values and don't prefix FAILED with our reserved prefix, I think that change is safe.

[TomHennen]

and to not have SLSA_BUILD_LEVEL_N imply SLSA_BUILD_LEVEL_N-1.

How would this change affect dependencyLevels?

If in dependencyLevels SLSA_BUILD_LEVEL_N doesn't imply SLSA_BUILD_LEVEL_N-1 then users can no longer detect when some dependency earlier in their supply chain violates their desire to be SLSA_BUILD_LEVEL_N.

[AdamZWu]

Oh I overlooked this comment. I agree with Tom -- since the SLSA BUILD track levels are defined in a stacked manner, I think "SLSA_BUILD_LEVEL_N imply SLSA_BUILD_LEVEL_N-1" makes sense?

[joshuagl]

It was suggested that explicitly stating each level verified may simplify policy evaluation. This could easily be something to consider in future. Let's scope this issue only to removing the FAILED value from SlsaResult

[ramonpetgrave64]

For the SLSA-track definitions, I think it appropriate to keep the N -> N-1 implication, but other tracks should be left open for more flexibility when implemented in a verifier tool, such as slsa-verifier.

Users MAY use custom values here but MUST NOT use custom values starting with SLSA_.

I'm wondering if VSA producers may have their own non-SLSA "tracks" that are not stacked in the same way that our SLSA tracks are, and maybe not appropriate to say that MY_LEVEL_3 implies MY_LEVEL_2.

I'd like to see a concrete example. But I imagine an alternative scenario: as policy, a consumer of a MY_LEVEL_2 artifact would want to refuse a MY_LEVEL_3 as artifact because the higher level is too sensitive for what the consumer is designed for.