Hot Reload for TLS Keystore

[Alwaysgone]

Hello everyone,

I haven't been able to find anything regarding hot reloading of TLS keystores during Quarkus runtime. I understand that Quarkus is pretty static in its concept, so dynamic reloading of resources while in the production profile is probably out of scope without custom code. Our current setup for TLS certificates is that we have short lived certificates, a few days vailidity, which are automatically reissued with newly generated passwords. Non Quarkus services based on Jetty have no problem reloading a newly issued keystore during runtime, we only had to write some glue code to detect keystore changes and then call the reload method on the SslContextFactory. It's probably important to note that the PKCS12 keystore is residing in an external config directory each Quarkus service accesses. So the service accesses a path like ../config/tls.p12 to get to it's keystore from inside an uber jar.

• Is it possible to achieve hot reloading in a currently relased Quarkus version? If yes how, we already have a trigger for when to reload, we would only need a means to do the reload itself.
• If it is not easily possible with Quarkus code alone, is there a way to implement this in a custom fashion?
• If it is a planned feature, it would be nice if the reload could be triggered with a CDI event or by simply calling a static method that handles the internals.

What's also important to note is that the keystore path itself does not change it is simply checked using file system events so the aforementioned ../config/tls.p12 is simply replaced and new passwords are pushed to the config.

Our current workaround for this is to simply call System.exit when a keystore change is detected and the service gets restarted automatically. This works but it does cause alerts in our monitoring and since it happens every few days, it makes it hard from distinguishing these deliberate restarts from actual crashes.

[Alwaysgone]

Sorry for the bump but it has been quite a while since this was opened. Any input on this would be greatly appreciated. Thanks in advance.

[tsaarni]

I'm also interested in this feature.

I think this could be enabled by vert.x. I've submitted PR https://github.com/eclipse-vertx/vert.x/pull/4372 to resolve issue https://github.com/eclipse-vertx/vert.x/issues/3780. I only implemented PEM files so far, not key stores, but I think same approach applies for key store files as well.

If the feature could be accepted by vert.x, we'd would need Quarkus to pass certificate & key as file paths, instead of passing them by file content which is done currently. Here is a small change that would do that https://github.com/Nordix/quarkus/compare/main...configure-certs-by-path.

[Hakky54]

Hi everyone, I have added my input regarding reloading the ssl configuration here: https://github.com/eclipse-vertx/vert.x/issues/4452 I am curious what everyones opinion would be for the solution.

[tsaarni]

It is now possible to implement hot-reload on the application side. Check #27481 and #27682.

[DMaxter]

Is there any example on how can we do this? I have a setup similar to @Alwaysgone where TLS key/certificate only live for a small amount of time and I would like Quarkus to detect a change in the files and just reload them. I read both the PR and the issue but didn't find a proper way to do this

[Hakky54]

I have a working example here: GitHub - Reloading SSL with Quarkus

It contains code snippets on how to provide your own ssl configuration and supply that to the underlying server (vertx) programatically. It also contains the tutorial of how to replicate it on your local dev environment. Basically you need to adjust your server configuration with a HttpServerOptionsCustomizer

import io.quarkus.vertx.http.HttpServerOptionsCustomizer;
import io.vertx.core.http.HttpServerOptions;
import io.vertx.core.net.KeyCertOptions;
import io.vertx.core.net.TrustOptions;
import jakarta.enterprise.context.ApplicationScoped;
import jakarta.inject.Inject;
import nl.altindag.server.service.FileBasedSslUpdateService;
import nl.altindag.ssl.SSLFactory;
import org.jboss.logging.Logger;

import java.nio.file.Path;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;

@ApplicationScoped
public class ServerConfig implements HttpServerOptionsCustomizer {

    private final Logger LOGGER;

    @Inject
    public ServerConfig(Logger logger) {
        LOGGER = logger;
    }

    @Override
    public void customizeHttpsServer(HttpServerOptions options) {
        var sslFactory = SSLFactory.builder()
                .withSwappableIdentityMaterial()
                .withIdentityMaterial(Path.of("/path/to/your/identity.jks"), "secret".toCharArray())
                .withSwappableTrustMaterial()
                .withTrustMaterial(Path.of("/path/to/your/truststore.jks"), "secret".toCharArray())
                .build();

        var sslUpdateService = new FileBasedSslUpdateService(sslFactory);

        Executors.newSingleThreadScheduledExecutor().scheduleAtFixedRate(sslUpdateService::updateSslMaterial, 1, 1, TimeUnit.MINUTES);
        LOGGER.info("Checking every minute for changes on the keystore and truststore files");

        options.setSsl(true)
                .setPort(8443)
                .setKeyCertOptions(KeyCertOptions.wrap(sslFactory.getKeyManager().orElseThrow()))
                .setTrustOptions(TrustOptions.wrap(sslFactory.getTrustManager().orElseThrow()));

        HttpServerOptionsCustomizer.super.customizeHttpsServer(options);
    }

}

This is the output which I get:

__  ____  __  _____   ___  __ ____  ______ 
 --/ __ \/ / / / _ | / _ \/ //_/ / / / __/ 
 -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \   
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/   
2023-05-12 03:58:14,069 INFO  [nl.alt.ser.ser.FileBasedSslUpdateService] (Quarkus Main Thread) Started listening for any changes on the keystore and truststore files...
2023-05-12 03:58:14,078 INFO  [nl.alt.ser.con.ServerConfig] (Quarkus Main Thread) Checking every minute for changes on the keystore and truststore files
2023-05-12 03:58:14,167 INFO  [io.quarkus] (Quarkus Main Thread) instant-server-ssl-reloading-with-quarkus 1.0.0-SNAPSHOT on JVM (powered by Quarkus 3.0.3.Final) started in 3.317s. Listening on: http://localhost:8080 and https://localhost:8443
2023-05-12 03:58:14,168 INFO  [io.quarkus] (Quarkus Main Thread) Profile dev activated. Live Coding activated.
2023-05-12 03:58:14,168 INFO  [io.quarkus] (Quarkus Main Thread) Installed features: [cdi, resteasy-reactive, smallrye-context-propagation, vertx]
2023-05-12 03:59:14,082 INFO  [nl.alt.ser.ser.FileBasedSslUpdateService] (pool-8-thread-1) Keystore files have been changed. Trying to read the file content and preparing to update the ssl material
2023-05-12 03:59:14,588 INFO  [nl.alt.ser.ser.FileBasedSslUpdateService] (pool-8-thread-1) Updating ssl material finished

In the example project I configured the server with an inital ssl configuration with an expiration date of somewhere in 2029. After updated the keystore it was shifted to 2031. The server is configured to scan the keystore files for any changes for every minute, however that can be changed to every hour, or every day or any other duration. The magic happens in the FileBasedSslUpdateService but that is not too difficult. It is just a service checking whether the keystores have been updated and if that is the case it will regenerate the SSLFactory and update the initial one.

[Alwaysgone]

For anyone else having this issue. We are just now getting around to refactoring this for our Quarkus services and the first version would be similar to what @Hakky54 has done but we will try to contribute this to Quarkus in the near future so it is either an extension or simply a set of properties for the https connector. Because we have a lot of internal tasks in the backlog I don't know when we can do this but I'm hoping this can be done until the end of this year.

[geoand]

Given that HttpServerOptionsCustomizer allows for this use case, do you folks think there is anything else to be done here?

[geoand]

#34997 will add something generic. We can certainly welcome contributions to that once it's in main 😎

[ahus1]

Happy to see some examples above. It would be great to see this functionality in Quarkus as other frameworks have it already today: https://spring.io/blog/2023/11/07/ssl-hot-reload-in-spring-boot-3-2-0/