SSL Support for TLS enabled Zookeeper

[techguy0079]

Hi,

I was trying to setup cruisecontrol for mutual TLS ZK Cluster and Kafka Cluster.

I have provided the ssl properties in cruisecontrol.properties , however when I run cruisecontrol startup script. I am getting issues when cruisecontrol tries to connect to ZK.

Can anyone clarify if SSL support for ZK exist in cruise-control-2.5.10 ? Kafka cluster - 2.5 ZK cluster - 3.5.7 (SSL ports :localhost:2281,localhost:2282,localhost:2283)

Error in ZK: [nioEventLoopGroup-4-6:NettyServerCnxnFactory$CertificateVerifier@434] - Unsuccessful handshake with session 0x0 2020-09-07 00:06:16,866 [myid:2] - WARN [nioEventLoopGroup-4-6:NettyServerCnxnFactory$CnxnChannelHandler@273] - Exception caught io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record:

Error in cruise control script: [2020-09-07 00:06:49,048] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn) org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server 'localhost:2281,localhost:2282,localhost:2283' with timeout of 60000 ms at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1233) at org.I0Itec.zkclient.ZkClient.(ZkClient.java:157) at org.I0Itec.zkclient.ZkClient.(ZkClient.java:131) at com.linkedin.kafka.cruisecontrol.detector.BrokerFailureDetector.(BrokerFailureDetector.java:66) at com.linkedin.kafka.cruisecontrol.detector.AnomalyDetector.(AnomalyDetector.java:110) at com.linkedin.kafka.cruisecontrol.KafkaCruiseControl.(KafkaCruiseControl.java:111) at com.linkedin.kafka.cruisecontrol.async.AsyncKafkaCruiseControl.(AsyncKafkaCruiseControl.java:34) at com.linkedin.kafka.cruisecontrol.KafkaCruiseControlApp.(KafkaCruiseControlApp.java:43) at com.linkedin.kafka.cruisecontrol.KafkaCruiseControlMain.main(KafkaCruiseControlMain.java:38)

Let me know if any additional info is needed.

[efeg]

Hi @techguy0079 Does the wiki on Secure zookeeper configuration help?

[techguy0079]

Hi @efeg , Yes I went through the link above, but as I am using mutual TLS only for ZK, I don't have configuration to mention as I am not using Kerberos authentication etc. The below entry for example doesn't help. What should be the entries into this file if I am only using mutual TLS ZK and not anything else. //Enter appropriate Client entry for secured zookeeper client connections Client { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/path/to/zookeeper_client.keytab" storeKey=true useTicketCache=false principal="zookeeper_client@<REALM>"; };

Regards techguy0079

[akatona84]

Kafka 2.5.0 supports TLS enabled ZK, CC is creating the kafka Zookeeper client (KafkaZkClient), here we should pass the ZKClientConfig as the last param: https://github.com/linkedin/cruise-control/blob/migrate_to_kafka_2_5/cruise-control/src/main/java/com/linkedin/kafka/cruisecontrol/KafkaCruiseControlUtils.java#L526

[akatona84]

@techguy0079 have you tried to pass ssl related zk config as system properties? these would be the basic props:

-Dzookeeper.client.secure=true \
-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty \
-Dzookeeper.ssl.keyStore.location=<ks location> \
-Dzookeeper.ssl.keyStore.password=<ks password> \
-Dzookeeper.ssl.trustStore.location=<ts location> \
-Dzookeeper.ssl.trustStore.password=<ts password>

Here you can see some others if required: https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/KafkaConfig.scala#L301

[techguy0079]

yes I tried those system properties too, but it doesn't help either.

[jrevillard]

Dear @techguy0079, @akatona84 , do you have a solution for this ? I'm facing the same problem.

Best, Jerome

EDIT: I just figured out that using the java properties listed works ! I was trying to set it inside cruisecontrol.properties originally. Thanks !

[Arullaldivakar]

@jrevillard ,

Is it working for you ? Can you please provide a solution. I have tried adding the above listed java system properties, but its not working.

[jrevillard]

Yes it is.

[Ownercz]

Can confirm, embedding options into the startup script for ZK works.

[jlevy8]

Using cruise-control-2.5.77 Kafka cluster - 2.8.1 ZK cluster - 3.5.9

Confirmed that passing settings in as JVM Parameters works (Properties file still does not work despite changes in https://github.com/linkedin/cruise-control/pull/1703)

However i am seeing an intermittent Len error in Zookeeper after enabling zookeeper.client.secure=true

java.io.IOException: Len error 369296129
	at org.apache.zookeeper.server.NettyServerCnxn.receiveMessage(NettyServerCnxn.java:536)
	at org.apache.zookeeper.server.NettyServerCnxn.processMessage(NettyServerCnxn.java:363)
	at org.apache.zookeeper.server.NettyServerCnxnFactory$CnxnChannelHandler.channelRead(NettyServerCnxnFactory.java:266)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:829)

[Phani2811]

Hello @techguy0079 - Were you able to fix the issue. am also facing same issue. Zookeeper is configured with mTLS. cruise control not able to communicate with zookeeper. can you pls help what needs to be passed below for TLS connections. Thank you.

//Enter appropriate Client entry for secured zookeeper client connections Client { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/path/to/zookeeper_client.keytab" storeKey=true useTicketCache=false principal="zookeeper_client@"; };