yahoo
CMAK should stop using Zookeeper as Kafka connection endpoint
Kafka is phasing out Zookeeper. All actions in Kafka are done via Kafka API and don't require direct connection to Zookeeper.
However CMAK still relies on defining Zookeeper as connection endpoint.
In our case Zookeeper is locked down (on network ACLs level), there is no direct connection to that from CMAK or anywhere else. Only from Kafka brokers.
Yep, we'll phase it out but we'll need some place to store metadata. Either directly in Kafka or RDBMS. Still to be designed.
I don’t mean that CMAK shouldn’t use Zookeeper to store it’s own metadata. But it should not require zookeeper info to connect to Kafka cluster.
Currently to add cluster it requires zookeeper connection string. While Kafka has already deprecated using zookeeper in most of their tools and clients. The cluster discovery and Kafka communication API shouldn’t involve Zookeeper.
Where CMAK stores it’s own metadata has no connection to this whatsoever. In our case we are running CMAK with its own zookeeper.
As a matter of fact as a user of CMAK I would highly prefer if it didn’t store any of its own data in Kafka. Because CMAK is a supporting tool. It shouldn’t pollute data in Kafka with stuff that’s not relevant there.
In our case Zookeeper is locked down (on network ACLs level), there is no direct connection to that from CMAK or anywhere else. Only from Kafka brokers. @akamensky How are you able to run CMAK with the ACL restriction to Zookeeper since the zookeeper is the connection endpoint for the cluster in CMAK?
@srikalam well, that is the point. We can’t after setting up the restrictions. Information is stale in CMAK and doesn’t get updated.
@akamensky Thanks for the confirmation. We are also facing the same issue on our usecase. Hopefully CMAK get updated soon to configure Kafka Cluster using brokers instead of zk hosts
The same for us: for security reason kafka zookeeper isolated from other services. It will be great to have ability to use Kafka api than direct zookeeper connection
@akamensky We are having the same issue and we agree with you that Kafka shouldn't be polluted with CMAK metadata either. I think en internal KV store should be introduced for metadata and all connection should be directed toward broker.
We run kafka version 3.2.3 without zookeeper (using kraft). Unfortunately, it seems that we can't use CMAK (which we consider very useful and convenient) anymore due to absence of zookeeper. Please, add support for kraft in newer kafka versions.
I think we should discuss this further and agree on a approach, CMAK is a very powerful tool and is widely used.
We can still use a ZK to store CMAK configuration and only change connectivity to Kafka cluster for cluster specific information. This will also reduce development work in the initial phase as think to have a different storing place for CMAK configuration.
Using and managing a standalone ZK should be doable and many organisations do it for other use cases including Kafka.
Thoughts ?
Actually we can store CMAK configuration in a dedicated Kafka topic ( compacted ? )
+1 for the option to store CMAK metadata in a Kafka topic. Now that KRaft is production-ready (as of 3.3.1), having to continue running ZK just for CMAK to be able to manage my clusters seems like a step backwards.
Currently to add cluster it requires zookeeper connection string. While Kafka has already deprecated using zookeeper in most of their tools and clients. The cluster discovery and Kafka communication API shouldn’t involve Zookeeper.
+1 for this feature
Hello, I want to use CMAK, but not using Zookeeper. +1 for this issue. Thank you.
+1 to this feature.
I'm also looking forward to use CMAK with Kraft.
