slsa-framework
[docs] clarify which build level one will reach when using a provided GitHub Action
NOTE: this issue was initially posted on the https://github.com/slsa-framework/slsa-github-generator repo and transferred here without modification.
Hello :wave:
Not really an issue but rather a question on the docs who might require a clarification (at least for newbies like me trying to understand stuff related to SLSA). It would have been probably more suited for GitHub discussions, but AFAICT these are not enabled in this repository :confused:
TL;DR:
- when using one of the GitHub Action provided in this repository, it's not clear which SLSA Build level one will reach
- by just using one of the provided builder in a GitHub Action provided here (let's take the Go builder as a concrete example), can I tell consumers of my artifacts that I have achieved L3?
Please read the details below though please :sweat_smile: :point_down:
Details
1. Why does "all of the requirements" are not met?
The "What is slsa-github-generator" section of the README states (emphasis mine):
While slsa-github-generator can help you achieve SLSA Build level 3, use of the provided GitHub Actions reusable workflows alone is not sufficient to meet all of the requirements at SLSA Build level 3.
I'm not sure I really understand this because when reading SLSA Build L3 level requirements, it seems (to me) that using a provided GitHub Action on a GitHub runner (let's take the Go builder as a concrete example) fills every single one of them, so it would be sufficient to meet all of the requirements (correct me if I'm wrong):
- Software producer follows a consistent build process so that others can form expectations about what a “correct” build looks like. :heavy_check_mark: (L1)
- Provenance exists describing how the artifact was built, including the build platform, build process, and top-level inputs. :heavy_check_mark: (L1)
- Software producer distributes provenance to consumers, preferably using a convention determined by the package ecosystem. :heavy_check_mark: (L1)
- Build platform runs on dedicated infrastructure, not an individual’s workstation, and the provenance is tied to that infrastructure through a digital signature [1]. :heavy_check_mark: (L2)
- Downstream verification of provenance includes validating the authenticity of the provenance. :heavy_check_mark: (L2)
- Build platform implements strong controls to:
- prevent runs from influencing one another, even within the same project. :heavy_check_mark: (L3)
- prevent secret material used to sign the provenance from being accessible to the user-defined build steps. :heavy_check_mark: (L3)
2. Are "provenance distribution" and "verification" part of the requirements?
Also, regarding the following sentence in the docs:
Specifically, these workflows do not address provenance distribution or verification. You can use the slsa-verifier to verify the provenance.
To me:
- the first part of the sentence regarding distribution is false: provenance is distributed using the GitHub Action: for example, here, with the Go builder, the provenance attestations are automatically uploaded to the release alongside the binary
- the second part regarding verification is unclear: SLSA spec is actually talking about verification, but it does not seems to be a requirement. It's up to the users to verify artifacts, so if verification is a requirement, I can't see how can a developer could ensure people downloading the artifacts also verify them :thinking: Maybe adding something in the docs to help users verify could help this, but there is no guarantee people will read these docs :smile:
3. Some contradictions?
Moreover, the whole last part of the README's "What is slsa-github-generator" section kinds of contradict what we can find in the builders or generators section:
They let you meet the provenance generation and isolation strength requirements for SLSA Build level 3 and above.
Or even a few lines above:
Specifically, this repository contains:
- tools for generating non-forgeable SLSA provenance on GitHub for your projects. The generated provenance meets the provenance generation and isolation requirements for SLSA Build level 3 and above.
I understand that here, it is just talking about provenance generation and isolation, and I'm aware there are more parts in the SLSA Producting artifacts Build levels:
But even then, it seems to me that using one of the provided builders check these boxes anyway:
- GitHub Actions is an appropriate build platform
- the build process is consistent: for example, when using the Go builder, build process will always be the same. I don't think "consistent" means "reproducible" here though
- provenance is distributed (see above)
Am I wrong?
Conclusion
As a developer, I wish I could easily find by reading the docs, when just using one of the provided builders:
- what SLSA Build level I will achieve?
- if I can display this badge
on my repository (if the answer to the previous question is level 3 obviously! :sweat_smile:)
Also, maybe this part needs to be clarified (or removed?) as it confused me a lot, so I'm assuming it can confuse other people:
While slsa-github-generator can help you achieve SLSA Build level 3, use of the provided GitHub Actions reusable workflows alone is not sufficient to meet all of the requirements at SLSA Build level 3. Specifically, these workflows do not address provenance distribution or verification. You can use the slsa-verifier to verify the provenance.
Thank you very much for reading, and looking forward to read your replies :pray:
Hi! 👋
Thanks for your question! I'll try to answer your questions individually.
1. Why does "all of the requirements" are not met?
I think you mostly answered this yourself, but we are mostly addressing the SLSA requirements placed on the "builder" rather than the overall requirements. So distribution and verification are out of scope, though I tend to agree that distribution is covered by most of the workflows via GitHub releases or npm in the case of the Node.js builder.
2. Are "provenance distribution" and "verification" part of the requirements?
Verification is included in the SLSA Build L2 requirements (SLSA Build L3 includes all requirements from L1 and L2).
Downstream verification of provenance includes validating the authenticity of the provenance.
This implies that the provenance is distributed somehow. I'd agree that the distribution docs read more like recommendations than requirements, but verification requires that you distribute the provenance in a way that the consumer can use.
3. Some contradictions?
I'm not sure I see the contradictions in the documentation. We specifically mention the provenance generation and isolation requirements, while verification needs to be implemented via another tool (slsa-verifier for artifacts/npm packages, npm for npm packages, Kyverno or sigstore-policy-controller for containers etc.).
In general, I tend to agree. The requirements for SLSA levels have been simplified quite a bit since we originally wrote the documentation. Currently we get you most of the way there.
SLSA covers up to and including the usage of software so if you're an OSS developer the best you can do is produce and release the software in an SLSA Build L3 compliant manner. i.e. build and provide provenance meeting the relevant requirements. It's up to consumers to verify to fully comply with the remaining requirements of the framework.
That said, do you have any specific wording suggestions for the docs? Maybe we can omit the bit about distribution?
@laurentsimon @MarkLodato thoughts?
I completely agree with the premise of this question. The status quo is too confusing. This is an issue with the SLSA Spec in general, beyond this specific repo.
I think it really comes down to question 2: Is "verification" part of the requirements?("Provenance distribution" is just a recommendation.) In other words, if you generate SLSA provenance in a manner that meets all the L3 requirements, but then nothing actually verifies it, does that satisfy SLSA Build L3?
This is something I've been struggling with all along. On the one hand, I feel like if no one checks it, you're not really preventing any tampering attacks. On the other hand, it's helpful for the producer to be able to just say "This is SLSA Build L3" without having some carefully worded caveat.
Maybe the solution is better terminology.
Shall we move this issue to the main https://github.com/slsa-framework/slsa/issues repo for visibility and to loop in others?
@norbjd you can tell your users your builds reach level 3. Provide an example how to verify the build with slsa-verifier in your README. You can also add a link on your release notes pointing to the README (GitHub release notes are not tamper proof, so it's better to provide the verification command in your GitHub README).
@ianlewis I think we should simplify our README to reduce confusion.
Shall we move this issue to the main https://github.com/slsa-framework/slsa/issues repo for visibility and to loop in others?
SGTM
@norbjd you can tell your users your builds reach level 3. Provide an example how to verify the build with slsa-verifier in your README. You can also add a link on your release notes pointing to the README (GitHub release notes are not tamper proof, so it's better to provide the verification command in your GitHub README).
Practically speaking, I think this is right. Producers should be able to say they are SLSA Build L3 if they meet the relevant requirements and provide some guidance on how to verify.
Transferred to the main SLSA repo. @MarkLodato feel free to loop in relevant folks.
Thank you very much for your answers! I think the "What is slsa-github-generator?" section (the most confusing for me) is somewhat clearer now. But, I wish I could have found what you said in the comments in the README though, as these really make sense! :smile:
From my point of view, as a simple user, even with the additional clarifications provided here, this part on the generator repo is still a little overwhelming. The main "selling point" of the builders is to allow developers to comply to the SLSA framework easily, thus making it accessible. I'm pretty sure a large majority of developers won't even bother reading the whole spec to be honest :grin: To make the documentation more appealing, we could:
- simplify it and go straight to the point. I think that's fine to just add a link to the main specification for the most curious users
- add practical and concrete examples
Here is my attempt based on all your comments (thanks again for clarifying, it was super helpful). Note that I'm not an english native speaker, so wording can be weird, but my attempt here is more about the "idea" than the definitive paragraph to add in the slsa-github-generator repo :sweat_smile:
SLSA consists of 3 parts:
- Build: generate the artifacts and their provenance attestations
- Provenance distribution: distribute the provenance attestations generated in the build step, so consumers can find it
- Provenance verification: check that the provenance is authentic: it's up to the consumers to run this verification
All builders here provide SLSA Build Level 3.
Provenance distribution is covered by most of the builders, via GitHub releases (or npm, Maven Central, etc.). [1]
Provenance verification is up to the consumers. However, as a producer, providing in your documentation the exact verification command to run, as well as a link to slsa-verifier, might help your users to verify your artifacts. [2][3]
For more details, check out the SLSA specification.
[1] For this one, I think we might also add a column "distribute provenance" in this table, just with :heavy_check_mark: or :x:. However, I'm not sure if it's useful because it looks like all builders are distributing provenance anyway. Am I right?
[2] Including a concrete example here (like https://v2-3.docs.fluxcd.io/flux/security/slsa-assessment/#example) might be helpful. Instead of relying on third-party repositories, maybe we could add a "How to verify?" section in the example-package README, and link it here!
[3] Maybe there are other tools than slsa-verifier, but I think that's fine to only focus on one (just my 2 cents). Obviously, unless some attestations generated by some builders can't be verified with this tool.
Tell me what you think.