ostree
ostree copied to clipboard
ostreedev
WIP: GPG remote trusted key updating
This is a work in progress that I started a while back and would like some comments on. In order to securely update the trusted GPG keys for a remote, it adds an implementation of the OpenPGP Web Key Directory client side. This lets a remote publish updated GPG keys at a known location and have ostree fetch them to update its local keyring.
There are still parts that need some polish, but I believe it mostly works. I'm mostly posting this to get feedback. The impetus for this work is that at Endless we had one of our trusted keys expire, which would have prevented people from updating. We had a workaround by having a 2nd key in the trusted keyring, but we would have preferred a reliable way to push an update for the expiring key.
[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: dbnicholson
To complete the pull request process, please assign cgwalters after the PR has been reviewed.
You can assign the PR to them by writing /assign @cgwalters in a comment when ready.
The full list of commands accepted by this bot can be found here.
Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment
@dbnicholson: PR needs rebase.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
For Fedora(host systems) we use gpgkeypath=/etc/pki/rpm-gpg/ - so keys are delivered as part of the ostree commit itself.
For Fedora Flatpak it's currently via OCI which...hm, I don't think we're GPG verifying right now.
For flatpak in general, wouldn't it be a lot simpler to sign .flatpakrepo files?
To be clear I'm not opposed to this, I just want to be sure that the simple solutions aren't sufficient.
For Fedora(host systems) we use
gpgkeypath=/etc/pki/rpm-gpg/- so keys are delivered as part of the ostree commit itself.
This is the same as Endless - key updates are delivered in OS updates via /usr/share/ostree/trusted.gpg.d (which is safer than something in /etc that might not survive the merge, but I digress). The trusted path to key updates is the key itself since you can't receive new keys without the key that verifies the update.
What happens when the OS update is no longer signed by that trusted key because it's expired or is compromised? There needs to be an out of band mechanism to update the local trusted key. Currently that out of band mechanism would be manual and technical - you'd send out a bunch of emails and make a web page explaining the situation with the new trusted key and instructions on how to add it to the local key store. This is a pretty high barrier. It requires the user to be looking for that type of information and to know how to handle a security sensitive process correctly. Turning that manual process into an automated process is the goal of this work.
For flatpak, there's already a mechanism for delivering keys via the xa.gpg-keys metadata key in the summary file. However, since flatpak remotes by default have summary GPG verification, I think it suffers from the same situation as above. If the remote summary needs to be signed by a new key because the existing one can no longer be trusted, then the client won't be able to verify it to receive updated keys and manual intervention will be needed. It felt to me more appropriate to deliver the keys out of band than through the repository itself. I.e., use a different trusted path to update this trusted path. I also was interested in a neutral method that could live in libostree to be used by any ostree application.
Just for some other background, here's a page on the gnupg wiki discussing different key update methods - https://wiki.gnupg.org/OpenPGPEmailSummit201607/KeyDiscoveryComparison. There are some other pages on the wiki discussing these types of things.
Lastly, I sent an email to the mailing list in https://mail.gnome.org/archives/ostree-list/2021-January/msg00000.html if you want to have a more general discussion there.
What happens when the OS update is no longer signed by that trusted key because it's expired or is compromised?
If it's compromised...I don't see how we can do anything? I need to dig into this a bit but what's the trust model for the web service? Is it just "TLS with keys from ca-certificates"?
This WKD stuff seems to mostly be about authenticating the email address in GPG keys, which seems orthogonal?
I guess let me say it this way: You've added a lot of new code here but I am missing a brief document that describes how one might set this up and use it. And a specific question: if ostree had had this support originally when your key expired, would it have saved you? How?
I feel like this key expired case is actually a special case of a more general one where one wants an "out of band hotfix" process for an ostree-based system. I believe Firefox has something like this - a mechanism they can use for truly critical fires.
A "hotfix system" wouldn't need to be implemented in ostree at all really, it could be a separate systemd service that pulls content however it wants and validates it integrity however and basically supports live-updating the system.
Of course, the question is how the integrity of those patches is validated...and if you're signing those correctly then presumably you're also signing your ostree commits correctly.
Can you split out the non-WKD patches to a separate PR?
Sure. No promises on when I'll get to that, but there are some things in here that are not specifically for WKD.
What happens when the OS update is no longer signed by that trusted key because it's expired or is compromised?
If it's compromised...I don't see how we can do anything? I need to dig into this a bit but what's the trust model for the web service? Is it just "TLS with keys from ca-certificates"?
If it's compromised but you can easily (or automatically) fetch an updated key from a trusted location, then you can carry on. For example, suppose my ostree signing key got compromised. I can generate a new key that has the same ID (email address) and publish it on a web server in a WKD compliant location. The client checks for an updated key at that location, sees there's something new, updates the local keyring and we carry on. It's basically the same process as if I'd written an email saying where to get my new key from and what to do with it except that it can happen automatically because the URL is predictable.
The trust model is TLS here and the fact that the key can only be fetched from a fixed URL within the domain it's email address is a part of. To spoof that process, an attacker would need to send a fake DNS reply and provide a valid TLS certificate.
This WKD stuff seems to mostly be about authenticating the email address in GPG keys, which seems orthogonal?
The email address determines the URL where the update can come from, so I was trying to be a bit paranoid about ensuring that was done correctly. This goes in 2 directions - determining the URL to make the request to and then only taking the key for that desired email address in the received keyring.
I guess let me say it this way: You've added a lot of new code here but I am missing a brief document that describes how one might set this up and use it. And a specific question: if ostree had had this support originally when your key expired, would it have saved you? How?
Sure, I can write something up on how you'd do it. https://wiki.gnupg.org/WKDHosting basically explains how you'd populate the server side, but I'll follow up with something more concrete.
It definitely would have saved us. After updating the expiration date on our key, we would have published the new key on our WKD server. Then the clients would fetch the updated key and signatures for our commits would be valid again.
$ /usr/lib/gnupg/gpg-wks-client --print-wkd-url "EOS OSTree Signing Key 1 (EOSK1) <[email protected]>"
https://openpgpkey.endlessm.com/.well-known/openpgpkey/endlessm.com/hu/e5t1hbkongpswdkafkdzkxwwxokkammg?l=maintainers
I feel like this key expired case is actually a special case of a more general one where one wants an "out of band hotfix" process for an ostree-based system. I believe Firefox has something like this - a mechanism they can use for truly critical fires.
A "hotfix system" wouldn't need to be implemented in ostree at all really, it could be a separate systemd service that pulls content however it wants and validates it integrity however and basically supports live-updating the system.
They feel a little different to me. Being able to provide a hotfix because the OS we published is itself broken or compromised sounds very useful. We have more than once released commits that could no longer boot or update and the only thing we could resort to was publishing some instructions for people to run manually. But this issue isn't with the OS itself, it's with the trust path. Hotfixing the payload vs hotfixing the delivery mechanism seems different to me.
That said, there isn't any requirement for this to be in ostree. You could have a service that occasionally checks for and updates the trusted keys. The nice thing about having it in ostree is that any ostree application can make use of it without depending on an outside service.
The trust model is TLS here and the fact that the key can only be fetched from a fixed URL within the domain it's email address is a part of. To spoof that process, an attacker would need to send a fake DNS reply and provide a valid TLS certificate.
I wouldn't assume any security with DNS in general, so this is basically TLS. And if you're trusting that absolutely...what's the point of GPG signing in the first place over just fetching the commit over that same TLS connection?
I wouldn't assume any security with DNS in general, so this is basically TLS. And if you're trusting that absolutely...what's the point of GPG signing in the first place over just fetching the commit over that same TLS connection?
Because they validate different things. TLS validates the transport while GPG validates the payload. Having separate validation of the payload means that you can use an untrusted transport such as any non-TLS network protocol or a local filesystem. Like would happen for peer to peer distribution. Likewise, payload verification means that if someone can insert a rogue commit into the remote repo it would still fail to be validated by clients.
Why do you sign the ostree git tags? Why did you develop git-evtag? Presumably we're all using a secure transport to access the remote repo (HTTPS or SSH), so by the same logic we can assume that any object in the git repo is valid. How do I get your public key to validate the signature on those tags? I wouldn't take it from an object in the repo and then use it to validate a tag on the repo. I would get it from some other trusted source.
TLS is also the security mechanism used for the transport of the trusted keys here, but I don't think that TLS also being the typical repo object transport security mechanism really has any bearing. They're trying to secure and validate access to different things even though they happen to use the same mechanism in this work. Indeed I am trusting TLS here just like we all do every day on the internet and just like the WKD developers determined would be acceptable.
I'm not arguing against GPG signing, you are right transport and offline integrity are different things.
I've glanced though the WKD stuff but haven't done a deep dive - it may be somewhere there and I'm just missing it. It seems to mostly be a protocol about discovery, leaving the "how to choose to trust this key" outside of the scope.
You are here basically automatically trusting keys found this way, right? So that to me seems to negate the whole point of using GPG in the first place if we unconditionally load new keys over TLS.
Perhaps one middle ground is that we only try to fetch keys from WKD in the case of expired keys by default? And in that case, require the new key be cross-signed from the old one?
Sorry for taking so long to respond.
I've glanced though the WKD stuff but haven't done a deep dive - it may be somewhere there and I'm just missing it. It seems to mostly be a protocol about discovery, leaving the "how to choose to trust this key" outside of the scope.
You are here basically automatically trusting keys found this way, right? So that to me seems to negate the whole point of using GPG in the first place if we unconditionally load new keys over TLS.
In the case of ostree, we're saying that we already trust the IDs in the keys for the remote's trusted keyring. You or someone on your behalf has already imported my key with [email protected] to trust my repo at the remote URL. What WKD does is give you a URL to find a new version of my key that's at a fixed URL tied to my domain. So, in case my key expires or I need to revoke it there's a known place to get it from that can't really be spoofed unless I lose control of my domain or TLS breaks down.
$ /usr/lib/gnupg/gpg-wks-client --print-wkd-url [email protected]
https://openpgpkey.endlessos.org/.well-known/openpgpkey/endlessos.org/hu/9q46ttg995y1xz6dq5fsheb18u177pz4?l=dbn
So, it's not totally arbitrary, but it does preclude that you trust the IDs that are in your trusted keyrings. Perhaps there could be a remote flag that says whether the key can be updated.
Perhaps one middle ground is that we only try to fetch keys from WKD in the case of expired keys by default? And in that case, require the new key be cross-signed from the old one?
I think revoked is the bigger issue, which you can't tell by looking at the key you currently have. I think cross signing could work. Another option would be that the primary key fingerprint is unchanged. In which case you'd do all your key management on subkeys.
Perhaps there could be a remote flag that says whether the key can be updated.
That's really what I'm focusing on - in exactly what circumstances can a key be updated via this mechanism, and what does it require?
If we're focusing on the expired key problem, then requiring new keys to be cross signed seems like a good default.
Another option would be that the primary key fingerprint is unchanged. In which case you'd do all your key management on subkeys.
Right; this is a common pattern in the GPG world for this.
Here's another random idea though, what if we patched ostree to warn loudly if a key it verifies is e.g. going to expire in a month or even 3 months? I mean honestly the "time bomb" aspect of this is a huge problem with all long term keys. Fixing this problem is really a lot of the benefit of Let's Encrypt - they force everyone to rotate keys frequently. This seems like a good thing to recommend in general even for GPG. Fedora generates a new key for each Fedora major every ~6 months, though we currently still trust even quite old ones.
Is it okay to rely GPG keys lookup on SHA-1 algorithm used in protocol? Shouldn't it be updated to more secure algorithms? It does not have support for different hash algorithms it seems to me.
Sorry I keep forgetting to reply on this.
For the expiration warnings, I think they could be a little useful. On the server side I probably would have noticed that before the expiration and done something about it. On the client side I can't see it really being useful, though. The warnings wouldn't be seen unless a CLI was in use, and even if they were users can't do anything to fix the issue. Still wouldn't do anything about a key that needed to be revoked, though.
I think that the really scary part here is the fully automated part. So, maybe let's not do that right now. If ostree can just help you see what keys are there and what the WKD URLs are, then anyone could essentially write their own automation around it. Providing a way to fetch and filter the returned key would be nice, but not really necessary. That's really not different than what someone could do now manually - figure out the email address from a remote's key, turn it into a WKD URL, download it and import it back into ostree. What I'm proposing now is having ostree just make parts of that easier to do. I think your initial "hey can you split out these general parts" would cover most of it. Whether someone wants to trust the WKD URL itself or what's returned from downloading it can be someone else's business.
I cleaned up and broke out the non-updating parts to #2401. I think with that in place the actual key updating and associated policy and security decisions can be made by someone else.
@dbnicholson: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:
| Test name | Commit | Details | Required | Rerun command |
|---|---|---|---|---|
| ci/prow/sanity | 6d46edfa9afa16b7dc708aec2520885729d237c4 | link | true | /test sanity |
Full PR test history. Your PR dashboard.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.
![openshift-ci[bot] avatar](https://avatars.githubusercontent.com/in/91280?v=4 "Published by a pub.dev verified publisher"){kind=small}