hashicorp
Signing boxes with pgp
It would be nice if it was possible to sign boxes with PGP when they are uploaded. Or have the disk images inside the box signed and the signature part of the box container. Vagrant could automatically or on request try to validate boxes.
@dragetd I've been pushing for this for years! My goal was to start embedding the signatures when the base box is built, by adding support to packer then focus on giving Vagrant the ability to verify those signatures... but there wasn't much interest at HashiCorp for it, and makes a bit of sense, since most of their big users are enterprise customers who manage box images internally using other tools.
The would really be for people who consume 'public' boxes, off something like the Vagrant Cloud. But if you look at Vagrant Cloud, there are literally just a few which routinely publish/update images, and are maintained by people with the skills to setup/use signing keys. And while those boxes probably account for most of the downloads, the reverse is true for most of the box images. As most are uploaded by individuals for personal projects, and those folks wouldn't know/care of about signing keys.
Anyways, my idea was to push the idea in person at HashiConf. I figured support for signatures would make a great hackathon goal. And that perhaps HashiCorp could organize the hackathon to coincide with HashiConf... but I wasn't able to attend in 2019, and there hasn't been an in person gathering since.
Not all is lost though, @chrisroberts added support to the Vagrant Cloud for box hashes. I don't know how many projects use the feature, but at least with the Robox images, I submit the hash along with the box file. And Vagrant will verify the file it downloads against that hash. Before that was implemented, box files would often get corrupted, such as during a partial download, or some other random ephemeral error. I did a lot of testing back in 2019, I was getting around 1 failure for 2,000 box pulls (most of the failures were fatal though, so Vagrant didn't try to use the image). As you can imagine, it started eating up bandwidth (I initally though GCP<->AWS bandwidth was gratis, but was wrong). So I don't know things have improved.
In terms of uploads. When I push a Robox release, I still get 1-3 upload corruptions per release. But that's out of 758 box files, being pushed at 10 gigs. On the plus side though, with the hashes, vagrant will reject the download when this does happen.
But on the negative, the Vagrant Cloud still wasn't validating/rejecting uploads which fail to match the preimage commit hash. See here ... https://github.com/hashicorp/vagrant_cloud/issues/74
Software supply-chain attacks are becoming more common over the years. I think this feature is very important.
@federico-razzoli I meant to reply right away, and started to, but got pulled away. What I wanted to ask 3 weeks ago was whether you, or @ar @chrisroberts @azr @dragetd would be at cdCon in Austin, which os taking place this week. But I realized while writing that reply, that all of the HashiCorp folks were likely to in SF this week for the RSA conference.
So, I'll ask instead, is anyone attending the Open Source Summit in a couple of weeks? It's also in Austin, and it's taking place at the same time as a couple of other Linux Foundation conferences, including 3 focused on security -which includes the SupplyChainSecurityCon! How appropriate, right?
I was thinking this feature is perfect for a hackathon. If we had enough of the right people in the room, we could get an initial, MVP implementation done in just 1 or 2 days. It would be easier that way, in part because doing this right will require, at a minimum Packer and Vagrant plugins that work in concert with each other. We'll beed a Packer that can be used by the Robox, Bento, et al projects, so we can sign the base boxes we generate. But we'll also need, at a minimum, a complimentary Vagrant plugin which uses thoses signatures. Namely, storing the mebedded certificates, and verifiying the signatures. That functionality alone would be a huge step forwrd, in my opinion, because it would be future box updates/downloads would be authenticated automatically. And it would allow large enterprise devops teams, and the security aware folks like myself, to embed those repo keys in Vagrantfiles manually. Which would ensure even the initial download is authentic.
Long term, a a more functionality will be needed. Off the top of my head, I'm thinking large enterprises, and those groups who create the most frequently used box images, will probably want to switch to storing their private keys inside an HSM. And we'll also need support organization keys, key rotation, and escrow signing, all of which would, ideally, require Vagrant Cloud support/integration. The Vagrant plugin shoild also have the ability (eventually), to sign deriviative box files before uploading them.
I was asked to write down my thoughts on this feature a long time ago., but the task got buried because nobody else seemed interested in working on this at the time, but if enough people can gather later this month in Austin to work on this (and I have enough notice), I'll commit to having a design doc ready for us to work off of during the hackathon.
If we don't have enough interest, we can try again at the next in-person HashiConf, like the event in Oct, in LA. Not sure if that counts as in person, or if I'll attend, but it seems like a natural place for people to gather and work on this feature.
P.S. I still can't believe the great and powerful @SwampDragons are no longer committing to the packer repo. You will be missed.
@ladar yours is surely a great idea. I couldn't develop it myself, not even an MVP. But I'd like to understand some things. You say that an MVP can be developed in 2 days - how many persons should work on it? The other question is - I understand that Vagrant is being refactored in Go (good, it means that it's not considered dead by Hashicorp!), and Packer is written in Go already, so I suppose that the feature should be entirely developed in Go?
I ask these things because my company, Vettabase, could be interested in sponsoring this feature.
@federico-razzoli I didn't know Vagrant was being refactored into Go, but I've spent all the time I have for the Robox project on maintaining/adding to the box configs. That takes up so much time that I haven't had much left over to keep up with changes to the code base. But I consider being out of touch a good thing in this case, because it means I haven't had any recent critical issues that required a code change to fix.
For a hackathon, I figure you'd need someone to take the lead on making changes to Packer, and another to focus on the Vagrant code changes. Then to make things go quickly/smoothly, a third person to audit the code, and make sure things work end to end. But having a couple of extra people to help with user documentation, unit tests, along with implementing optional features, would be ideal. You'd need more time, or more programmers if the leads I mentioned weren't already familiar with the Packer/Vagrant code bases. I figure a single expert in Go/Ruby who already knows cryptography/PGP, but who isn't intimately familiar with the Packer/Vagrant code base (presumably they know how to use the tools though) could probably build this in 2-4 weeks, depending on how through they are with unit tests, etc. The Vagrant/Packer code base knowledge is part of the reason I think we'd need at least 1 person from HashiCorp in the room, who presumably has commit access (more on this below). At the very least we'd need someone whose spent a lot of time creating/working on code/plugins for each tool, and someone from HashiCorp whose willing to review/merge the pull request.
Personally, I've made small changes to Packer and Vagrant, but that was awhile ago, and I don't work with Go (or Ruby for that matter) often enough to consider myself an expert in those languages. That's a big part of why I haven't tried to write this myself already... I could certainly help with the other tasks though.
As for MVP. It would probably look something like this:
1.0 = Packer would gain the ability to generate and embed a signature file, along with the public key in a box tarball. Vagrant would gain the ability to verify the signatures, and track/store the public key it encounters. It would use a TOFU model by default, but users could add a repo public key fingerprint to their Vagrantfiles ahead of time to avoid issues with a TOFU model. Something akin to config.vm.box_download_checksum but unlike that feature, this new parameter would allow someone to update a box without having to manually edit the Vagrantfile each time. Finally, Vagrant would need to remove the signature and public key whenever it repackages a box.
1.5 = Vagrant would get a new sub command to facilitate managing stored keys, along with the ability to sign boxes with a new public key if they were repackaged, and uploaded to a cloud store for others to use.
2.0 = Vagrant Cloud integration would introduce org level keys, which would be signed by Vagrant Cloud, and could in turn sign individual repo keys*. That would allow Vagrant to verify the chain when it first encounters a new repo key, and eliminate some of the risks associated TOFU. At this point, Vagrant should also be able to handle downloads signed by multiple keys, which would facilitate key rotations.
2.5 = Vagrant Cloud could offer an optional key escrow feature, whereby auto-generated public keys could be created for every repo, and this escrow could be used to automatically sign box uploads from users who don't want to manage their own signing keys. Presumably a user could decide whether they want to manage the key, or rely on escrow when creating a new repo. And such uploads would have to use the traditional Vagrant Cloud upload method, and not the direct upload method, so this signing could take place. If a user changes their mind later, they can convert the repo (which would require enforcing rules which require some discussion) but presumably converting the repo from one type to the other, would involve download (or uploading) the current private key, and along with the requirement that the key be rotated. The upload/download step is to facilitate a smooth rotation, but couldn't be used as the new "current" key.
- Technically speaking, someone could use the same public key for multiple repos, and then later, use that same key again as the org key, but that isn't required.
I don't think any of this can occur without HashiCorp's blessing though. While it would be possible to write a Packer plugin that signed a virtual hard disk, after the build stage, and the current post processor for Vagrant allows including an arbitrary file in a tarball (I think), or at the very least would allow embedding the signature/public key as part of the bundled info JSON file, this wouldn't ideal because some box types include more than a virtual hard disk, and I believe with some, the files aren't known/created until the Vagrant post processor is called. Which means making it work properly would require extending that existing post processor. Building it this way means, and maintaining it, if it weren't accepted upstream, would be a chore. So knowing HashiCorp was onboard ahead of time seems like a per-requisite, and another reason it didn't seem worth spending the 2 weeks it would take me to create an MVP, if all it would did was create another ongoing chore for me.
On the Vagrant side, the signature check step should occur after a download fingerprint is verified, and the tarball is exploded. I assume there is an extension point at this stage for a standalone plugin, but don't know for sure. So technically this part doesn't require HasiCorp's blessing, and it might make sense to start this way. But over the longer the long term, a plugin like this would need to be bundled with the official releases, or it would be hard to balance the required effort against the benefits.