Confluent helm charts not working with community Kafka

[mjschmidt]

I know that metrics is now enterprise only.

Hi I am looking to use the open source kafka helm chart. I set jmx in the cp-kafka chart to false and changed the image from enterprise to the basic cp-kafka:5.2.2, however when I helm install my kafka dies with a class not found exception. My question is what other yaml configurations need to be made to this values file to get kafka to not look for metrics? link below to the values file I am looking at.

thanks! https://github.com/confluentinc/cp-helm-charts/blob/master/charts/cp-kafka/values.yaml charts/cp-kafka/values.yaml

# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

## ------------------------------------------------------```
 Show more
confluentinc/cp-helm-charts | Added by GitHub
the class that it cannot find is io.confluent.metrics.reporter.ConfluentMetricsReporter. I would think if I turned off jmx metrics exporter it shouldn't look for this class?

[emy-lee]

Hi, I have the same issue. How did you solve? Thanks

[mjschmidt]

I did not. I still run an older version of the helm chart and instead I just uptic the version of Kafka in there. I believe i am using helmchart 1.0.0? But I think Confluent might be overwriting 1.0.0 helm chart? idk i'll let that to these folks to comment on.

[OneCricketeer]

ConfluentMetricsReporter reports data to ControlCenter via Protobuf, not over JMX, BTW, so turning off one would not remove the exception.

You would need to disable/remove the metrics reporters variables in the respective images.

https://github.com/confluentinc/cp-helm-charts/blob/master/charts/cp-kafka/values.yaml#L57

[mjschmidt]

ah just disable the jmx variable? I don't think that was there when I was last deploying! Good to know that got addressed now

edit: not the jmx just make that variable blank*

[OneCricketeer]

No. The JMX isn't related to the error you've posted. You must set the metrics reporter environment variables to be empty strings

[mjschmidt]

yup, I see that now. Sorry I responded too quickly thank you for the guidance that is awesome.

[OneCricketeer]

Welcome! Happy Kafka-ing 😊

[mjschmidt]

so, how would i set this from the helm install? It looks like I have to helm fetch and untar and change specifically that values file for it to work. Shouldn't this be configurable from the higher level helm chart?

[OneCricketeer]

Something like --set customEnv='{"KAFKA_METRICS_REPORTERS": "" }'

https://helm.sh/docs/topics/chart_template_guide/values_files/

[mjschmidt]

right understood... but you can't set that param using the higher level helm values file since its a nested chart structure? Wouldn't the higher level chart have to have that value option for that to work? Maybe I am wrong here?

[OneCricketeer]

Right, you must add the override section, similar to #310

[scoquelin]

For the record, this is how I got it to work with Helm 3 and version 0.4.1 (I believe it should work for 0.5.0 as well) using --set from the parent chart :

--set cp-kafka.customEnv.KAFKA_METRIC_REPORTERS=

so using Helm 3 installing the community version for 0.4.1 could potentially translate into :

helm install my-confluent-oss --set cp-control-center.enabled=false --set cp-kafka.image=confluentinc/cp-kafka --set cp-kafka.customEnv.KAFKA_METRIC_REPORTERS= confluentinc/cp-helm-charts --version 0.4.1

[paul-bormans]

Making metrics support an enterprise feature is a complete surprise to us and actually i think it's incorrect.

The ConfluentMetricsReporter (packaged in confluent-metrics-X.Y.Z.jar) is only included in the enterprise image. The issue of the crash (class not found) is caused by the chart setting this reporter as default. So either commenting or overriding it does the trick.

Having said that i don't know yet if w/h ConfluentMetricsReporter you actually get metrics out at all over JMX, but i assume yes, since on older (5.4.0) community image this also worked.

Paul

[joseph-skupniewicz-sp]

We ran into this issue during a Kafka upgrade and it led to a significantly longer time to deploy/upgrade because the cp-kafka Helm chart does not offer any way to disable ConfluentMetricsReporter, and this is enabled by default, and we don't (yet) run the enterprise version of cp-kafka, and the crash error message was not very clear or useful:

2022-11-12 00:56:22,774] INFO [Controller id=0] Starting the controller scheduler (kafka.controller.KafkaController)
[2022-11-12 00:56:22,789] ERROR [KafkaServer id=0] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.kafka.common.KafkaException: Class io.confluent.metrics.reporter.ConfluentMetricsReporter cannot be found
	at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstance(AbstractConfig.java:374)
	at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstances(AbstractConfig.java:443)
	at kafka.server.DynamicMetricsReporters.createReporters(DynamicBrokerConfig.scala:795)
	at kafka.server.DynamicMetricsReporters.<init>(DynamicBrokerConfig.scala:744)
	at kafka.server.DynamicBrokerConfig.addReconfigurables(DynamicBrokerConfig.scala:247)
	at kafka.server.KafkaServer.startup(KafkaServer.scala:356)
	at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
	at kafka.Kafka$.main(Kafka.scala:82)
	at kafka.Kafka.main(Kafka.scala)
Caused by: java.lang.ClassNotFoundException: io.confluent.metrics.reporter.ConfluentMetricsReporter
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:398)
	at org.apache.kafka.common.utils.Utils.loadClass(Utils.java:410)
	at org.apache.kafka.common.utils.Utils.newInstance(Utils.java:399)
	at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstance(AbstractConfig.java:372)
	... 8 more

We ultimately needed to comment out lines 131-134 in the cp-kafka statefulset to disable this, as there is no way to readily disable this feature in the helm chart.

Request: Please find a better way to allow the installation to move on if ConfluentMetricsReporter is not available, or please provide a way to disable this in the Helm chart.