Custom Kafka connect jmx config

[ducminhle]

Hi All,

When I use cp-helm-charts, I use our own kafka-connect images, it builds from image confluentinc/cp-kafka-connect but we install some plugin from Confluent hub (for example Debezium-MySQL, Debezium-PostgreSQL, ...). We need to get metrics (example: Debezium) we need to change configmap Kafka connect jmx, but cp-helm-charts didn't support this feature. Could we add a feature custom Kafka connect jmx config for cp-helm-charts? Thank you

[OneCricketeer]

Can you give an example of what metrics you're interested in and what configurations you're trying to modify that cannot be placed in your image?

[ducminhle]

Dockerfile:

FROM confluentinc/cp-kafka-connect:6.1.0
RUN confluent-hub install --no-prompt debezium/debezium-connector-postgresql:1.4.0
RUN confluent-hub install --no-prompt debezium/debezium-connector-mysql:1.4.0
RUN confluent-hub install --no-prompt confluentinc/kafka-connect-s3:5.5.3

About metrics I customize jmx config to:

jmx-kafka-connect-prometheus.yml: |                                                                                                                                                                        
     jmxUrl: service:jmx:rmi:///jndi/rmi://localhost:5555/jmxrmi                                                                                                                                              
     lowercaseOutputName: false                                                                                                                                                                               
     lowercaseOutputLabelNames: false                                                                                                                                                                         
     ssl: false                                                                                                                                                                                               
     rules:                                                                                                                                                                                                   
     - pattern : "kafka.connect<type=connect-worker-metrics>([^:]+):"                                                                                                                                         
       name: "cp_kafka_connect_connect_worker_metrics_$1"                                                                                                                                                     
     - pattern : "kafka.connect<type=connect-metrics, client-id=([^:]+)><>([^:]+)"                                                                                                                            
       name: "cp_kafka_connect_connect_metrics_$1_$2"                                                                                                                                                         
     - pattern : "kafka.connect<type=connector-task-metrics, connector=([^:]+), task=([^:]+)><>status: ([^:]+)"                                                                                               
       name: "cp_kafka_connect_connect_connector_metrics"                                                                                                                                                     
       value: 1                                                                                                                                                                                               
       labels:                                                                                                                                                                                                
         connector: $1                                                                                                                                                                                        
         task: $2                                                                                                                                                                                             
         status: $3                                                                                                                                                                                           
     - pattern: "debezium.([^:]+)<type=connector-metrics, context=([^,]+), server=([^,]+), key=([^>]+)><>RowsScanned"                                                                                         
       name: "debezium_metrics_RowsScanned"                                                                                                                                                                   
       labels:                                                                                                                                                                                                
         plugin: "$1"                                                                                                                                                                                         
         name: "$3"                                                                                                                                                                                           
         context: "$2"                                                                                                                                                                                        
         table: "$4"                                                                                                                                                                                          
     - pattern: "debezium.([^:]+)<type=connector-metrics, context=([^,]+), server=([^>]+)>([^:]+)"                                                                                                            
       name: "debezium_metrics_$4"                                                                                                                                                                            
       labels:                                                                                                                                                                                                
         plugin: "$1"                                                                                                                                                                                         
         name: "$3"                                                                                                                                                                                           
         context: "$2"                                                                                                                                                                                        
     - pattern: 'java.lang<type=Memory><(\w+)MemoryUsage>(\w+): (\d+)'                                                                                                                                        
       name: jvm_memory_bytes_$2                                                                                                                                                                              
       labels:                                                                                                                                                                                                
         area: "$1"                                                                                                                                                                                           
       value: $3                                                                                                                                                                                              
       type: GAUGE                                                                                                                                                                                            
     - pattern: "java.lang<type=Threading><>(\\w+)"                                                                                                                                                           
       name: "jvm_threads_$1"

[hpsony94]

Hi,

How is it going on? same question. This ticket is to ask for a way to custom jmx-kafka-connector container which is sidecar pod as long with kafka-connector pod.

As you can see here: https://github.com/confluentinc/cp-helm-charts/blob/3ffbdf93dad1baf8a3c9a58a92b8e44bb848cd1c/charts/cp-kafka-connect/templates/jmx-configmap.yaml#L11

There is fixed prome declarative configuration, we need more flexible to get such as debezium metric.

Any update, soon? :) pls Thanks so much

[OneCricketeer]

custom jmx-kafka-connector container which is sidecar

You cannot reach JMX beans from a sidecar without exposing them though something like Prometheus or Jolokia from the Java process itself.

That being said, the ConfigMap is not "fixed"; it can be set in your own Chart definition

[alex88]

This would be awesome because we could use this dashboard https://github.com/confluentinc/jmx-monitoring-stacks that has way more data in it than the current default dashboard