Potential Leak of OpenSslKeyMaterial

[rpless]

I've started seeing a Netty Error LEAK: OpenSslKeyMaterial.release() was not called before it's garbage-collected. on a Finagle HTTP service.

Expected behavior

I would expect it to not leak the OpenSslKeyMaterial.

Actual behavior

Once or twice a day, I see a message in the logs that looks like this: https://gist.github.com/rpless/505395bebcb15367d2e287ae621354af

Steps to reproduce the behavior

I'm using Finagle 18.9.0 currently, but I don't see anything SSL related in the changelog between then and now. I'm currently configuring the server with the following options:

val keyCreds =  KeyCredentials.CertKeyAndChain(???, ???, ???)
val sslConfig = SslServerConfiguration(keyCredentials = keyCreds)
Http.server.withTransport.tls(sslConfig)

One thing I noticed about this service is that one of Finagle HTTP clients it uses generates a large number of ChannelClosedExceptions. The service that these clients are querying is known to crash randomly or timeout to requests, however the Leak warning doesn't happen immediately after a channel closed exception (sometimes its hours later). Not sure how that could be related because the OpenSslKeyMaterial is being generated from com.twitter.finagle.netty4.ssl.server which suggests its the server stack having the issue and not the client stack. I'm also not 100% sure if this is causing a problem. Our telemetry doesn't indicate anything leaking or failing as a result of this, so I'm not really sure what the impact is. I haven't been able to reproduce this locally and I haven't seen this in any of our other Finagle services. I have some thoughts on a potential way to reproduce if it is indeed caused by this interaction with another service, but it will take me little time to get a minimal case.

[kevinoliver]

This is unexpected. I haven't seen any reports of this internally, but that doesn't mean it isn't happening. If you can reproduce, you could turn on Netty's more detailed leak detection to ADVANCED (https://netty.io/wiki/reference-counted-objects.html#wiki-h3-11) and get the backtrace.