akka
Wrong TLS session after ssl handshake completed over TLSv1.3 protocol
I have a REST API that is authenticated with client certificates. The application is implemented using akka-http (akka version 2.6.18, akka-http version 10.2.7), java runtime: openjdk version "11.0.14" 2022-01-18 LTS.
I am constantly seeing failures in acquiring the proper SSL session after the SSL handshake was successfully completed.
For instance, enabling the SSL debug sequence - the handshake was completed, the client certificate was trusted and the session was successfully initialized:
javax.net.ssl|ALL|1B|m-a-p-system-akka.actor.default-dispatcher-5|2022-05-11 09:05:14.733 CEST|SSLSessionImpl.java:242|Session initialized: Session(1652252714355|TLS_AES_256_GCM_SHA384)
When I'm trying to fetch (using an akka directive the session, the code snippet below), sometimes the SSL session is the wrong one:
[info] [2022-05-11 09:05:14,786] INFO a.a.ActorSystemImpl - [] Tls session: Tls-Session-Info: Session(1652252714209|SSL_NULL_WITH_NULL_NULL)
[info] [2022-05-11 09:05:14,789] ERROR a.a.ActorSystemImpl - [] Error by getting peer certificates due to javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
[info] [2022-05-11 09:05:14,933] INFO c.h.a.m.r.ApplicationRoutes - [] Headers List(Timeout-Access:
I can reproduce it regardless if the client is using curl, akka http client or okhttp client from Kotlin.
I provide an implementation for trusting the client certificate according to java security specifications, and after the client is trusted, I need some information from the certificate itself, using akka directive:
val certificateChain: Directive1[CertChain] = {
for {
tlsInfo <- headerValueByType[`Tls-Session-Info`](`Tls-Session-Info`)
logger <- extractLog
certificates <- {
logger.info(s"Tls session: ${tlsInfo.toString()}")
Try(provide(tlsInfo.getSession().getPeerCertificates)) match {
case Success(certs) => certs
case Failure(exception) =>
logger.error(s"Error by getting peer certificates due to {}", exception)
...
}
}
As you can see, sometimes the session retrieved by headerValueByTypeTls-Session-Info is not the initialized session in the SSL handshake, but rather a new one.
Checking our production logs, out of 15000 requests 500 are failing with this error.
Last note: I can reproduce it only over TLSv1.3 protocol, if I use TLS 1.2 it seems to be fine. But that's not an option for me :(.
Thanks for your detailed report - this sounds rather serious. I have tried to reproduce it in isolation (https://github.com/raboof/akka-http-4122) but so far have not yet been able to. Could you see if that project reproduced the problem on your system? ~I have not yet set up mTLS there.~ I set up mTLS (though cheating a bit by using the same certificate for both server and client). I did check it's using TLS 1.3. Can you spot the difference with your scenario?
Hi @raboof, thanks for your quick feedback. I will check your project the soon as I have a chance and get back to you with feedback.
So far, I wasn't able to reproduce it with your prototype, but of course, our application is a bit more complex. I will try to keep adding more stuff to your example and see if I can manage to reproduce it in the end.
I will try to keep adding more stuff to your example and see if I can manage to reproduce it in the end.
That is great, thanks!
I have to correct my statement: the issue is not reproducible using curl (at least not with the latest version available on MacOS Monterey 12.3.1). I tried both: your prototype and my application with more than 20000+ requests, but no occurrence, the session was always properly retrieved from the Tls-Session-Info header.
However, the issue is still visible in:
- my integration tests, where we are using akka-http client. The above logs are from running the integration tests (User-Agent: akka-http/10.2.7)
- our production logs:
[2022-05-13 13:37:25,455] INFO c.h.a.m.r.ApplicationRoutes - [] Headers List(Timeout-Access:
, Remote-Address, Tls-Session-Info: Session(1652449045090|SSL_NULL_WITH_NULL_NULL), Accept: /, X-Correlation-ID: Host, Connection: Keep-Alive, Accept-Encoding: gzip, User-Agent: okhttp/4.9.0). In this case the client is implemented using okhttp 4.9.0.
Maybe the problem is on the client-side and the implementation is not properly covering some corner cases (e.g. session resumption over tls1.3), I can't tell.
I also found this ticket: https://github.com/akka/akka-http/issues/3920 which somehow seems to be related.
And again, at least for our integration tests, if I switch the protocol version to tls 1.2, I can't reproduce it anymore.
Thanks for the investigation!
I also found this ticket: #3920 which somehow seems to be related.
That sounds like potentially the same issue indeed - would be great if we could get to the bottom of it.
However, the issue is still visible in:
* my integration tests, where we are using akka-http client. The above logs are from running the integration tests (User-Agent: akka-http/10.2.7) * our production logs: [2022-05-13 13:37:25,455] INFO c.h.a.m.r.ApplicationRoutes - [] Headers List(Timeout-Access: , Remote-Address, Tls-Session-Info: Session(1652449045090|SSL_NULL_WITH_NULL_NULL), Accept: _/_, X-Correlation-ID: Host, Connection: Keep-Alive, Accept-Encoding: gzip, User-Agent: okhttp/4.9.0). In this case the client is implemented using okhttp 4.9.0.Maybe the problem is on the client-side and the implementation is not properly covering some corner cases (e.g. session resumption over tls1.3), I can't tell.
OK, so the next step would be trying to reproduce it with an Akka HTTP client and the reproducer project at https://github.com/raboof/akka-http-4122 ?
That will be great, I will try to see on my side if I can quickly modify my integration tests to run against your application as well. I tried to modify your project, but I'm not using scala 3 and I ran into all sorts of weird errors when adding the scala-test library.
I converted https://github.com/raboof/akka-http-4122 to Scala 2 now, if that helps :)
I managed to reproduce it by adding an integration test with your prototype. I had to make some changes in terms of certificates to make it closer to our setup and not disable the hostname verification. Unfortunately, I can't share the certificates I used with you, but it will probably be trivial to replace them with yours (self-signed certificates). If you grant me access to your project, I can push my changes.
This is a snapshot of the output: Running server (sbt run):
[info] running (fork) Main [info] 10:18:00.771 [default-akka.actor.default-dispatcher-4] INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started [info] 10:18:01.257 [main] DEBUG main - Starting up [info] 10:18:01.284 [test-akka.actor.default-dispatcher-6] INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started [info] 10:19:01.695 [test-akka.actor.default-dispatcher-10] INFO main - SSL_NULL_WITH_NULL_NULL
Running the test (sbt "integration-tests/test"):
[info] CheckResponseTest: [info] Server [info] - should return success response *** FAILED *** [info] 500 did not equal 200 (CheckResponseTest.scala:18) [info] Run completed in 2 seconds, 621 milliseconds. [info] Total number of tests run: 1 [info] Suites: completed 1, aborted 0 [info] Tests: succeeded 0, failed 1, canceled 0, ignored 0, pending 0 [info] *** 1 TEST FAILED *** [error] Failed tests: [error] CheckResponseTest [error] (integration-tests / Test / test) sbt.TestsFailedException: Tests unsuccessful [error] Total time: 4 s, completed 17 May 2022, 10:19:01
I usually reproduce it after no more than 5 attempts.
Hope this helps.
I managed to reproduce it by adding an integration test with your prototype. I had to make some changes in terms of certificates to make it closer to our setup and not disable the hostname verification.
That's great to hear!
If you grant me access to your project, I can push my changes.
I invited you as a collaborator to the project
I am getting
git push --set-upstream origin add_test 128 ↵ 1164 11:04:47 remote: Permission to raboof/akka-http-4122.git denied to adibaranga. fatal: unable to access 'https://github.com/raboof/akka-http-4122.git/': The requested URL returned error: 403
You are still listed 'Awaiting adibaranga’s response' - I guess you need to accept the invitation somehow?
Did you manage to reproduce the issue?
Yes I have, thanks a lot! I have been looking into the issue, and the good news is I think there is no security issue: AFAICS the client behaves correctly, but the server indeed may pass the old, 'unauthenticated' session to the user code along with data from the 'authenticated' session. This is an upstream bug in Akka, now filed as https://github.com/akka/akka/issues/31419.
Great! Could you please drop me a message here in case of any progress for the upstream ticket? Thanks!
