mumble
mumble copied to clipboard
mumble-voip
Handle server-certificate renewals
Context If the server in question uses a certificate from e.g. Let's Encrypt, it is going to be renewed every 60 to 90 days. Every time that happens, the server will use a new certificate (the renewed one) that will generate a new certificate hash.
The effect of this is shown in #4516
Furthermore the new certificate is not picked up without restarting the server in the first place.
Describe the feature you have in mind The client should remember domains of servers it has been connected to before and what their cert-hash was. And every time the client connects to a server that presents an unknown cert-hash, it should go and check whether the certificate is signed for a domain that is already known to the client (due to having been connected to it before). In that case, it should assume that the certificate for the server has been renewed and offer the user with a prompt to migrate all data that is currently associated with the server's old hash to the new one.
On top of that the server should automatically use the new certificate after a successful renewal.
Additional context For externally signed certificates this certainly makes sense as we can trust the domain that is specified in the certificate (after successful verification of the cert), but for self-signed ones we probably shouldn't apply these changes.
Furthermore the situation in which a single murmur instance is hosting multiple virtual servers using the same certificate has to be considered. It has be checked how the client is currently coping with that situation and the new feature might have to be adapted accordingly.
See also https://github.com/mumble-voip/mumble/issues/5404
@sadsfae if I missed something, please let me know.
Thanks @Krzmbrzl for your work looking into this and potential for enhancement. One question so far as implementation, how would the mumble user prompt if cert-hash/settings were migrated? Would this appear for each user that connected that had prior data saved/associated to that old cert and cert-hash? e.g. Access Tokens?
Having a way to auto-migrate previous settings based on the combination of cert + cert-hash would be really nice I think as most people using Access Tokens do want to migrate their associative settings for a particular server.
From a client perspective accepting a prompt every 60-90 days (in case of Let's Encrypt) might get confusing for new users and old and repetitive for veterans, so it makes sense to have this as a configurable murmur.ini setting perhaps?
The prompt will probably show up as a simple pop-up window with a yes/no option. And it would probably pop up regardless of whether there is actual data that is to be migrated (though this can of course b controlled by the implementation).
Migrating by default without a prompt can be made available by a client-side setting. Turning this into a server-side setting will not happen though as this is none of the server's business (pure client-side decision).
Just for the record, it would be possible to have a mumble client pin on something else than the cert-hash. But that would make the ux completely different i guess.
Hm btw, does the client do any typical validation of cert though the chain, towards the root?
Just for the record, it would be possible to have a mumble client pin on something else than the cert-hash. But that would make the ux completely different i guess.
That would probably be possible but I guess having this tied to the cert somehow also makes sure that no server can really steal another's identity :thinking:
Hm btw, does the client do any typical validation of cert though the chain, towards the root?
I'm not entirely sure what you mean by that, but what I can say is that afaik the certificate validation is handled through Qt and we basically rely on them "doing the right thing" :eyes:
Proposed Solution:
One issue with a auto-migration based solution is that it has the potential to be vulnerable to domain sniping attacks. This is when someone steals a lapsed domain before the owner is able to renew it.
For externally signed certificates this certainly makes sense as we can trust the domain that is specified in the certificate (after successful verification of the cert), but for self-signed ones we probably shouldn't apply these changes.
In the case of this type of auto-migration, trusting that it's okay to migrate based on the fact that the certificate is valid for the corresponding domain and signed by a valid CA leaves us open to this attacks via vulnerability.
This vulnerability doesn't only affect domain sniping situations, it also can cause issues with service providers that reuse domains between clients. This situation has a much lower risk, as AFAIK unless the service provider gives users root access to the hosting machine, the access tokens are only exposed to the service provider itself, who already had access to them before the server was reused for a different customer.
Potential Solution
The server creates a "certificate chain". Each time a certificate is updated, the server would sign the new certificate with the public key of the previous certificate. When a user connects, the server would provide the entire chain of certificates. A client could then iterate through the chain and compare their last seen certificate to each element to see if they establish a chain of trust between some previous certificate and the latest certificate. If this is possible, I think it would be reasonably safe for the client to migrate access tokens forward.
Required Changes
Protocol
- Modify the handshake protocol to allow the server to present this new certificate chain while establishing a connection
- Optional: modify the handshake protocol to allow the server to tell the client what migration strategies are acceptable
Server
- Ingest the certificate into the database instead of reading it directly from disk on each launch (or rescan signal)
- Add logic to detect if the certificate file configured in
murmur.iniis different from the one in the database, and if so...- Sign the new certificate with the previous and add it to the chain
- Replace the certificate in the database with the new one
- Add support for resetting the certificate chain
- Add
murmur.inisupport for enabling/disabling this behavior, or... - Optional: add
murmur.inisupport for picking from a list of acceptable client migration behaviors - Support for handshake protocol modifications
Client
- UI support for migration (Yes / No prompt)
- UI support for auto-migration
- Add logic to evaluate the validity of the presented certificate chain
- Support for handshake protocol modifications
Backwards Compatibility
- The high level goals of this proposal seem to be backwards compatible with older clients, as they don't ever support migrating tokens.
- I've not thought enough about the technical details of how this would be implemented to determine what backwards compatibility would look like on a protocol level.
- Migrations would need to be added to the client and server as there would be required database changes on both sides.
Other Issues
-
This solution is of course still vulnerable to situations where an attacker obtains root access to the machine running murmur. I don't think this issue is something we can address here. It's probably better solved by something like https://github.com/mumble-voip/mumble/issues/1813
-
The optional migration strategies (and some of this feature in general) might be reasonably described as a false sense of security for server hosts. Users are still free to share their access tokens manually at will. I think we are striking a reasonably secure middle ground here, but maybe others have more thoughts.
-
There was some discussion around using expired certificates to validate a chain of trust. If there is a risk with using expired certificates, this would be a consideration as every certificate (except the current) in the chain would likely be expired.
Other notes
- Calling the "certificate chain" a "certificate chain" is a bit confusing as that name already has an existing meaning in the TLS space. We might want to pick some other name.
References
Related Matrix discussion: https://matrix.to/#/!tSdznLcpOUvPyMqEkQ:matrix.org/$AJCwCqgIpCQbzQtvbv2sjofEk6gAygR_-FMJPz60qVg?via=matrix.org&via=ohea.xyz
For Let's Encrypt users, I use the following workaround to restart murmur when a new certificate is deployed, otherwise murmur will continue to use the same (old, potentially expired) certificate that it has in memory.
(example assumes you're running murmur as a systemd service)
/usr/bin/certbot renew --cert-name www.mydomain.net --renew-hook "systemctl restart murmur" --quiet
This is less than ideal as depending on when the certificate is up for renewal or when you have cron check for it the restart can drop all logged in users.
At least with certificates generated against a browser/OS trust authority (like Let's Encrypt) after the service restart I believe there is no need for clients to acknowledge anything. For self-signed certificates there would be a client-side prompt every time those are replaced, although typically infrequent.
@sadsfae
this is less than ideal as depending on when the certificate is up for renewal or when you have cron check for it the restart can drop all logged in users.
If you want to avoid restarting the server, you can use https://github.com/mumble-voip/mumble/pull/2850 to reload the certificate.
@sadsfae
this is less than ideal as depending on when the certificate is up for renewal or when you have cron check for it the restart can drop all logged in users.
If you want to avoid restarting the server, you can use #2850 to reload the certificate.
That's intriguing, how would I reload the certificate with a letsencrypt hook if I may ask?
It's a bit off topic for this thread, but I would guess you would want to do something like
kill --signal SIGUSR1 <murmur_pid>
I'm not sure if you are running murmur with systemd or something else, but you will need to use something equivalent to systemctl [--user] show --property MainPID --value <service> to get the PID to pass to kill.
If you are only running one murmur instance, you could try using pkill and the process name instead and skip getting the PID altogether.
@sadsfae
this is less than ideal as depending on when the certificate is up for renewal or when you have cron check for it the restart can drop all logged in users.
If you want to avoid restarting the server, you can use #2850 to reload the certificate.
That's intriguing, how would I reload the certificate with a letsencrypt hook if I may ask?
Certbot takes care of all of this for you, including replacement. https://eff-certbot.readthedocs.io/en/stable/using.html#certbot-command-line-options
In my murmur configuration I point it to the LE symlink that gets updated when a new certificate drops so it's always using the right one, I had to modify the permissions to accommodate the murmur user for this to work.
sslCert=/etc/letsencrypt/live/www.mydomain.net/fullchain.pem
sslKey=/etc/letsencrypt/live/www.mydomain.net/privkey.pem
This is just a symlink that Certbot keeps updated to the lastest renewal set.
/etc/letsencrypt/live/www.mydomain.net/fullchain.pem -> ../../archive/www.mydomain.net/fullchain30.pem
/etc/letsencrypt/live/www.mydomain.net/privkey.pem -> ../../archive/www.mydomain.net/privkey30.pem
From here just reference a restart command in the --renew-hook command used with certbot, for example I use systemd so I use:
00 11,16 * * * /usr/bin/certbot renew --cert-name www.mydomain.net --renew-hook "systemctl restart murmur" --quiet
Thanks for the tip about https://github.com/mumble-voip/mumble/pull/2850
I need automatic reloading of the certificate without having to send a SIGUSR1 as I want to run this in a completely containerized way on Kubernetes. I've so far created a docker-compose file with Mumble and linuxserver/docker-swag, and SWAG is able to renew the cert without any cronjob. Now I just need Mumble to detect whenever the cert has been modified and automatically reload it.
See https://github.com/mumble-voip/mumble-docker/issues/28 for more details.
