Allow certificate rotation

[jroper]

Akka HTTP currently requires supplying an instantiated SSLContext at bind time. This is incompatible with certificate rotation, since when a certificate is rotated, a new SSLContext with the a keystore containing the newly rotated certificate needs to be created and used.

To support certificate rotation, Akka HTTP should allow supplying a function to create the SSLContext, and invoke that each time it needs an SSLContext. By default it may just return the same one each time, but users that want to support certificate rotation may provide a function that looks up and caches the context, and periodically reconfigures as needed.

[jroper]

Just had a look at implementing this, it's not as straight forward as it would seem, since the TLS flow in Akka streams also needs to be updated to accept a function to create an SSLContext, either that or Akka HTTP needs to be modified to create a new server flow on each connection.

[jrudolph]

Thanks, @jroper. That sounds like a useful feature.

Just had a look at implementing this, it's not as straight forward as it would seem, since the TLS flow in Akka streams also needs to be updated to accept a function to create an SSLContext, either that or Akka HTTP needs to be modified to create a new server flow on each connection.

There's an TLS.apply method that allows passing a function () => SSLEngine which is called once per connection. It was introduced because some custom SSL implementations (like the netty native one) don't support SSLContext but do support SSLEngine. It should be possible to use that one and back it by an SSLContext that can be changed when necessary.

On the other hand, we don't yet use this apply method in akka-http and don't provide the APIs to use it with HttpsConnectionContexts. How urgent would you need that functionality, so we can determine if we should work on this for 10.2.0?

[jroper]

For now we only have one service that needs it, it's using a letsencrypt certificate with 90 day expiry that gets rotated with 30 days remaining, so as long as that service gets restarted every 30 days, we'll be ok. Though that service is very simple and not likely to be updated very often. Nevertheless, manually restarting every 30 days is a possible work around, so it's not going to block us.

[jroper]

I've started work on this here, creating the public API for supplying the SSLContext creator, what's left to be done is to hook this up to the TLS flow, I may take a look at this myself or if someone else is going to take it up they may find my work a good starting point:

https://github.com/jroper/akka-http/commit/5af752b8e91588a8b1a6c075a98ea11855c70df8

[jrudolph]

I've started work on this here, creating the public API for supplying the SSLContext creator, what's left to be done is to hook this up to the TLS flow, I may take a look at this myself or if someone else is going to take it up they may find my work a good starting point:

jroper@5af752b

Thanks for sharing, not sure if we will get to it first but it looks like a good first step.

[SJern]

Hello, do we have any update on this? I have a similar certificate rotation requirement for Akka Management. For the Play server or Akka Remote, I could extend the SSLEngineProvider to make them reload once every day, but don't think I can do the same for Akka Management. By the way, I thought Play framework uses Akka http as the default server. How does the Play server make Akka http use the SSLEngineProvider? If the Akka http could indeed use the SSLEngineProvider, then we could extend the provider to refresh the SSLContext periodically.

[SJern]

@jroper what if the SSLContext you provided at bind time was a DynamicSSLContext which extends SSLContext?

[bblfish]

I find the letsencrypt certificate service really helpful for my 5 domains running on a single Apache server. As I would like to replace that with my Akka based web server Reactive-Solid I am also interested in this functionality being available for akka.

The letsencrypt protocol I believe is described in RFC 8555: ACME Protocol. Is it possible yet to implement this in Akka yet? (And potentially how difficult to implement it would be?)

I found the following projects:

• valters/play-acme-protocol not worked on in the past 5 years
• ACME4J Java library that seems active
• nginx-proxy was pointed to me by @raboof on gitter and I thought I'd just add this here for the record

[jroper]

While it's definitely possible to implement the ACME protocol in Akka, I don't think it makes sense to. Akka is a technology that is designed to be run across multiple nodes, for scaling and/or resilience purposes - there are many far simpler HTTP servers out there to use if you don't want Akka's clustering abilities. So, in a typical Akka deployment, you have many nodes. It doesn't make sense that each of those nodes would provision their own certificate using ACME, they should all share the same certificate. In fact you'd have a challenge doing it because each time a given node tries to provision a certificate, you need to ensure the challenge requests get routed to that node. Of course, that's possible to do with Akka cluster. But I think it makes more sense to use something external to Akka to manage the certificates. We use k8s cert-manager for this purpose. The reason I raised this issue is to support picking up certificates that have been newly rotated by k8s cert-manager, not just for the ACME use case, but moreso for a frequently rotated certificate use case, where you have certificates that have a 24 hour or shorter expiry (again, managed by cert-manager).

[Hakky54]

Hi guys, I noticed this discussion as I have been working on this feature for different servers like Spring and Netty for GRPC and the same configuration works for Akka too. I wanted to provide my input to this discussion as there is no need for any change on the Akka library needed in my opinion to get this working. The Akka server needs a configured SSLContext. The SSLContext holds a reference to a KeyManager and TrustManager which can be updated at runtime whenever needed, either by triggers or by a schedule or calling it manually. I am updating the SSL configuration at runtime with the following snippet:

import akka.actor.ActorSystem
import akka.http.scaladsl.server.Directives.complete
import akka.http.scaladsl.server.{Directives, Route}
import akka.http.scaladsl.{ConnectionContext, Http, HttpsConnectionContext}
import nl.altindag.ssl.SSLFactory
import nl.altindag.ssl.util.SSLFactoryUtils

import java.nio.file.Paths
import java.util.concurrent.{Executors, TimeUnit}
import javax.net.ssl.SSLContext

object AkkaReloadableSslerver {

  def main(args: Array[String]): Unit = {
    implicit val system = ActorSystem()
    implicit val dispatcher = system.dispatcher
    
    // Create dummy ssl configuration which can be swappable// 
    val baseSslFactory = SSLFactory.builder
      .withDummyIdentityMaterial
      .withDummyTrustMaterial
      .withSwappableIdentityMaterial
      .withSwappableTrustMaterial
      .build

    // Give this SSLContext to your akka server configuration
    val sslContext: SSLContext = baseSslFactory.getSslContext

    // A function to update the ssl configuration
    val sslUpdater: Runnable = () => {
      val updatedSslFactory = SSLFactory.builder
        .withIdentityMaterial(Paths.get("/path/to/your/identity.p12"), "password".toCharArray)
        .withTrustMaterial(Paths.get("/path/to/your/truststore.p12"), "password".toCharArray)
        .build

      SSLFactoryUtils.reload(baseSslFactory, updatedSslFactory)
    }

    // initial update of ssl material to replace the dummies
    sslUpdater.run()

    // update ssl material every day
    Executors.newSingleThreadScheduledExecutor.scheduleAtFixedRate(sslUpdater, 1, 1, TimeUnit.DAYS)

    val https: HttpsConnectionContext = ConnectionContext.httpsServer(sslContext)
    val routes: Route = Directives.get { complete("Hello world!") }
    Http().newServerAt("127.0.0.1", 8443).enableHttps(https).bind(routes)
  }

}

I have made this SSLFactory and SSLFactoryUtils available in this library: GitHub - SSLContext Kickstart I hope you guys like it, although I am referencing to my own library here..

[jroper]

We implement this today as @jrudolph suggested by using the () => SSLEngine method of constructing an HTTPS connection context:

            val contextReader = new RefreshableSSLContextReader(config)
            val ctx = ConnectionContext.httpsServer { () =>
              val engine = contextReader.getContext.createSSLEngine()
              engine.setUseClientMode(false)
              engine
            }
            Http()
              .newServerAt(bindAddress, port)
              .enableHttps(ctx)
              .bind(routes)

The RefreshableSSLContextReader implementation depends on your own requirements, but ours looks something like this, it loads the keys/certs from the filesystem, and caches them for a configured amount of time, reloading when expired:

class RefreshableSSLContextReader(config: RefreshableSSLContextReaderConfig) {

  import RefreshableSSLContextReader._

  private val rng = new SecureRandom()

  private val contextRef = new AtomicReference[Option[CachedContext]](None)

  def getContext: SSLContext =
    contextRef.get() match {
      case Some(CachedContext(_, expired)) if expired.isOverdue() =>
        val context = constructContext()
        contextRef.set(Some(CachedContext(context, config.refreshPeriod.fromNow)))
        context
      case Some(CachedContext(cached, _)) => cached
      case None =>
        val context = constructContext()
        contextRef.set(Some(CachedContext(context, config.refreshPeriod.fromNow)))
        context
    }

  private def constructContext(): SSLContext =
    try {
      val certChain = CertificateReader.read(new File(config.certFile))
      val caCertChain =
        config.caCertFile.toSeq.flatMap(file => CertificateReader.read(new File(file)))

      val keyStore = KeyStore.getInstance("JKS")
      keyStore.load(null)
      // Load the private key
      val privateKey =
        DERPrivateKeyLoader.load(PEMDecoder.decode(Files.readString(Paths.get(config.keyFile))))

      (certChain ++ caCertChain).zipWithIndex.foreach {
        case (cert, idx) =>
          keyStore.setCertificateEntry(s"cert-$idx", cert)
      }
      keyStore.setKeyEntry(
        "private-key",
        privateKey,
        "changeit".toCharArray,
        (certChain ++ caCertChain).toArray
      )

      val kmf = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm)
      kmf.init(keyStore, "changeit".toCharArray)
      val keyManagers = kmf.getKeyManagers

      val ctx = SSLContext.getInstance(config.protocol)
      ctx.init(keyManagers, Array(), rng)
      ctx
    } catch {
      case e: FileNotFoundException =>
        throw new RefreshableSSLContextException(
          "SSL context could not be loaded because a cert or key file could not be found",
          e
        )
      case e: IOException =>
        throw new RefreshableSSLContextException(
          "SSL context could not be loaded due to error reading cert or key file",
          e
        )
      case e: GeneralSecurityException =>
        throw new RefreshableSSLContextException("SSL context could not be loaded", e)
      case e: IllegalArgumentException =>
        throw new RefreshableSSLContextException("SSL context could not be loaded", e)
    }
}

object RefreshableSSLContextReader {
  private case class CachedContext(cached: SSLContext, expires: Deadline)
}