Handle Rotating TLS/SSL Certificates without Restart

[jkratz55]

This is more of a discussion/question than an issue. We are looking for a solution to be able to rotate our certificates for Kafka without restarting the application. We have so many microservices, most of which aren't using consumer groups for legacy architecture reasons, so it's an operational nightmare for us.

I haven't taken a deep dive into the code but since kafka.Dialer type has a pointer to tls.Config I was thinking about using fsnotify to watch the certificate files on the filesystem, and if a change is detected, update the tls.Config object the Dialer has a handle too. Then in theory (totally just guessing) when a new connection is established it will do so with the updated tls.Config.

But before even attempting to go down this road, I was curious if there is any support in the library or recommendations from the maintainers.

[dmarkhas]

AFAIK this is a long standing issue in the Kafka ecosystem (in all client languages).

I would suggest that instead of coding the rotation logic in the library, it should be externalized so the user can decide when / how to refresh the certificate, perhaps by placing a message on a channel..

On the other hand I do think this is a potentially risky behavior - to rotate the certificate you would need to close the connection, replace the dialer and reconnect, which could lead to data loss in the client as well as a "rebalance storm" if dozens of clients rotate their certificate simultaneously.

[petedannemann]

Hi @jkratz55, your proposal seems reasonable. I think after the certificate is rotated you'd need to close the Conn opened from the Dialer and then open a new connection. Externalizing this outside of the library as @dmarkhas suggested seems like the best way forward since this is fairly opinionated.