Server cert hot reloading

[dsyzhu]

Please answer these questions before submitting your issue.

What version of gRPC are you using?

v1.18.0

What did you expect to see?

Server can reload new cert without restarting

[dapengzhang0]

This totally depends on whether or not the io.netty.handler.ssl.SslContext object that you provide to io.grpc.NettyServerBuilder supports hot reloading.

[sanjaypujare]

To add to what @dapengzhang0 said, you will need to provide your own X509KeyManager or X509TrustManager (and possibly your own KayManagerFactory and TrustManagerFactory) that support hot reloading of certs/keys as per your requirement. Let us know if you are interested in implementing this.

[dsyzhu]

Can I mimic DelegatingSslContext to create a SslContext wrapper class and update the wrapped SslContext object when it is necessary?

[sanjaypujare]

@dsyzhu that might work and is worth trying. Although all you need is the Key and Trust Managers that support hot reloading, but your idea might be simpler. You have to be careful to not leak resources - I remember reading a warning somewhere in the source code in connection with SslContext.

[dsyzhu]

@sanjaypujare and @dapengzhang0,

Thanks for your prompt response.

I have a question of mimic io.netty.handler.ssl.DelegatingSslContext. The abstract class has an abstract fucntion:

     /**
     * Init the {@link SSLEngine}.
     */
    protected abstract void initEngine(SSLEngine engine);

What is it used for? Can I just give an empty implementation?

Thanks

[sanjaypujare]

@dsyzhu it depends on the concrete subtype of SSLEngine that is being passed. If that needs initialization then you can do it here. That's what I think - I have not played with SSLEngine nor DelegatingSslContext.

BTW the warning I was talking about is for io.netty.handler.ssl.OpenSslServerContext and io.netty.handler.ssl.ReferenceCountedOpenSslServerContext. You can check the source code on the netty repo.

[sanjaypujare]

@dsyzhu I am looking forward to your PR for this implementation :-)

[dsyzhu]

@sanjaypujare,

Thanks for information.

I will try to submit a PR.

[ejona86]

The KeyStore API doesn't really define whether it is thread-safe. But it seems like it has to be and both PKCS12KeyStore and JavaKeyStore (for JKS) use locks.

So I think I would suggest just changing the KeyStore live.

You'll want to specify KeyManagerFactory to SslContextBuilder. To create it, you'd do something like:

KeyStore ks = KeyStore.getInstance(KeyStore.getDefaultType());
ks.load(null);
CertificateFactory cf = CertificateFactory.getInstance("X.509");
X509Certificate cert = (X509Certificate) cf.generateCertificate(inputStream);
// If you don't want to hard-code principalName, you can use cert.getSubjectX500Principal().getName();
String principalName = "theserver";
ks.setCertificateEntry(principalName, cert);
KeyManagerFactory kmf = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
kmf.init(ks, null); // I'm not 100% about the null "password" here

And then later when you want to change the certificate, you'd use:

CertificateFactory cf = CertificateFactory.getInstance("X.509");
X509Certificate cert = (X509Certificate) cf.generateCertificate(newInputStream);
ks.setCertificateEntry(principalName, cert);

I'd want to double-check that KeyManagerFactory is compatible with netty-tcnative, but I think it may be. If not, then you just need to use Java's SSL provider instead of tcnative.

[sanjaypujare]

And if you want to change the private key (and associated public certificate) the code would be like this:

byte[] keyBytes = privateKey.getEncoded();
Certificate[] certChain = ...;
ks.setKeyEntry(principalName, keyBytes, certChain);

@dsyzhu if this thing works, do you still want to submit a PR?

[ejona86]

@sanjaypujare, a PR for what?

[sanjaypujare]

@ejona86 exactly. He may not need a PR - but I would like to hear what he says

[dsyzhu]

Our use case is to periodically reload cert before it is expired. I would focus on SslContextBuilder. Every time create a new SslContextBuilder object and call build() function to create SslContext to update SslContextWrapper class. Won't go deep inside SslContextBuilder. Any pitfalls if doing this way?

[ejona86]

@dsyzhu, the way I suggested seems to be less code and doesn't need re-creating the SslContext.

[yihuazhang]

@ejona86 Just to clarify, since an SslContext can be created by SslContextBuilder that takes KeyManagerFactory as an argument, each time the underlying KeyManagerFactory object gets refreshed with a new credential, the change will be automatically propagated the SslContex object, and thus there is no need to restart the server. Is it correct?

[ejona86]

@yihuazhang, yes, that's the expectation. When KeyManagerFactory is mutated (via it's KeyStore), the server will see those changes on future connections. Do test it, but it seems it should work and would be a limited amount of code to implement.

[yihuazhang]

@ejona86 SG. I will come up with a toy example to validate it.

I also found a thread that discusses the same issue (https://stackoverflow.com/questions/51411594/reload-netty-servers-ssl-context-for-grpc), but have different suggestions (also proposed by you Eric :)

[ejona86]

Yeah :). You can tell I was uncertain in that answer. I had seen that the KeyManager could change the alias, but I didn't think you could easily mutate the KeyStore (I had considered a DelegatingKeyStore, though). Noticing that KeyStore was mutable after init() was important, as well as figuring out whether KeyStore was thread-safe (technically the API doesn't say it is, but it sort of has to be, and the main implementations are... It's the same problem as HttpSession in the servlet API).

I had looked into it again for @sanjaypujare and had come up with this easier option. If things work out I'll add this approach to the SO question, because it is much easier.

[dsyzhu]

Thanks for all your inputs. Am wondering why server cert reloading hasn't been provided. I think all the grpc applications using tls connection need this feature.

[dsyzhu]

For my case I do something like:

public abstract class DelegatingSslContext extends SslContext {

private SslContext ctx;
private final SslContextBuilder ctxBuilder;

protected DelegatingSslContext(SslContextBuilder builder) throws SSLException {
    this.ctx = builder.build();
    this.ctxBuilder = builder;
}

@Override
public final boolean isClient() {
    return ctx.isClient();
}

@Override
public final List<String> cipherSuites() {
    return ctx.cipherSuites();
}

@Override
public final long sessionCacheSize() {
    return ctx.sessionCacheSize();
}

@Override
public final long sessionTimeout() {
    return ctx.sessionTimeout();
}

@Override
public final ApplicationProtocolNegotiator applicationProtocolNegotiator() {
    return ctx.applicationProtocolNegotiator();
}

@Override
public final SSLEngine newEngine(ByteBufAllocator alloc) {
    SSLEngine engine = ctx.newEngine(alloc);
    initEngine(engine);
    return engine;
}

@Override
public final SSLEngine newEngine(ByteBufAllocator alloc, String peerHost, int peerPort) {
    SSLEngine engine = ctx.newEngine(alloc, peerHost, peerPort);
    initEngine(engine);
    return engine;
}

@Override
protected final SslHandler newHandler(ByteBufAllocator alloc, boolean startTls) {
    SslHandler handler = ctx.newHandler(alloc, startTls);
    initHandler(handler);
    return handler;
}

@Override
protected final SslHandler newHandler(ByteBufAllocator alloc, String peerHost, int peerPort, boolean startTls) {
    SslHandler handler = ctx.newHandler(alloc, peerHost, peerPort, startTls);
    initHandler(handler);
    return handler;
}

@Override
public final SSLSessionContext sessionContext() {
    return ctx.sessionContext();
}

/**
 * Init the {@link SSLEngine}.
 */
protected abstract void initEngine(SSLEngine engine);

/**
 * Init the {@link SslHandler}. This will by default call {@link #initEngine(SSLEngine)}, sub-classes may override
 * this.
 */
protected void initHandler(SslHandler handler) {
    initEngine(handler.engine());
}

private synchronized void update() throws SSLException {
    ctx = GrpcSslContexts.configure(ctxBuilder).build();
}

}

GRPC server will use DelegatingSslContext and schedule a thread to call update() periodically or necessarily.

Am thinking this issue may be created for netty api

[creamsoup]

fyi. i updated the comment (just formatted the code snippet)

[dsyzhu]

Thanks

On Thursday, February 21, 2019, Jihun Cho [email protected] wrote:

fyi. i updated the comment (just formatted the code snippet)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/grpc/grpc-java/issues/5335#issuecomment-466232107, or mute the thread https://github.com/notifications/unsubscribe-auth/AKopxGxLtA67UiW7kx3Eu9p6u1xiVoFYks5vP0JzgaJpZM4aoyke .

[sanjaypujare]

@dsyzhu have you tested your code by any chance?

[sanjaypujare]

CC @jiangtaoli2016 @yihuazhang to comment on the proposal and code

[dsyzhu]

@sanjaypujare,

I did base function test. After I update cert I can the next connection will use the new cert and request succeeds. I can do more tests of launching a group of client thread to simulate real world traffic to do further verification. Do you think it is necessary? Or the code structure guarantees no hassle during cert switch?

BTW our use case is periodically refresh cert. So I use ScheduledThreadPoolExecutor to periodically call updateCert. That is why the function is private. For general cases it may be public to let other class to call it.

And I think it may be easier to make the change in netty.

Thanks

[jiangtaoli2016]

@dsyzhu We are designing a gRPC SSL utility library that can handle

• Credential reloading
• Root certificate reloading
• SPIFFE identity support
• Server authorization check plugin

Would you be interested?

In gRPC C core we are providing something similar via SPIFFE credentials.

[gava2727]

@jiangtaoli2016 I am interested in the library that automatically reloads the certs. Did you start working on it? do you have any timeline when it will be ready?

[magpor]

Hi

[magpor]

I am also interested in the util can you provide?

[sgaruda]

@dsyzhu I liked your idea of using DelegatingSSLContext. I have one question: So you will pass instance of DelegatingSSLContext while building the NettyServer right? Like below: Server server = NettyServerBuilder.forAddress(addr); builder.bossEventLoopGroup(bossEventLoopGroup); builder.workerEventLoopGroup(workerEventLoopGroup); builder.channelType(EpollServerSocketChannel.class); builder.maxInboundMessageSize(agentConfig.grpcMaxMessageSizeBytes); builder.maxHeaderListSize(agentConfig.grpcMaxHeaderListSizeBytes); builder.addService(ServerInterceptors.intercept(service, interceptors)); builder.sslContext(**instance of DelegatingSSLContext**).build();