snowflake-kafka-connector
snowflake-kafka-connector copied to clipboard
snowflakedb
Unable to get JMX metrics specific to Snowflake connector
Unable to get JMX metrics specific to Snowflake connector. Could you please share the sample JMX configuration for Snowflake connector. I couldnt find this anywhere.
I have tried but it's not working. Could you please share the sample JMX config file. Thanks
Confluent: https://docs.confluent.io/platform/current/installation/docker/operations/monitoring.html
Reference articel: https://docs.datastax.com/en/kafka/doc/kafka/monitoring/kafkaEnableJmx.html
I have tried with below pattern in my JMX config file.
- pattern: 'snowflake.kafka.connector<connector=(.+), pipe=(.+), category=(.+)><>(.+): (.+)' name: snowflake_kafka_connector_$3 labels: connector: "$1" pipe: "$2" category: "$3" "$4": "$5"
Additionally, PFB the config parameters in my dockerfile.
ENV KAFKA_JMX_OPTS="-javaagent:/usr/share/java/kafka-connect/jmx_prometheus_javaagent-0.17.0.jar=0.0.0.0:9280:/opt/optum/jmx-config.yaml -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
Still I couldnt find the JMX metrics mentioned as per snowflake documentation. Kindly help if there is any issue in the config.
can you try this:
- pattern: snowflake.kafka.connector<connector=(.+), pipe=(.+), category=(.+), name=(.+)><>(.+)
name: snowflake_kc_$3_$4
labels:
connectorName: $1
pipeName: $2
category: $3
metric: $4
type: GAUGE
Also do you use any ui based software like jconsole or visualvm to first identify if such mbeans are reporting the metrics?
https://grafana.com/blog/2020/06/25/monitoring-java-applications-with-the-prometheus-jmx-exporter-and-grafana/
You should be able to go to localhost:
Hi @sfc-gh-japatel ,
Sorry for late response on this thread. I have tried above config. However, still i am not getting any snowflake specific metrics.
- pattern: "snowflake.kafka.connector<connector=(.+), pipe=(.+), category=(.+), name=(.+)><>(.+)" name: snowflake_kc_$3_$4 labels: connector: $1 pipe: $2 category: $3 metric: $4 type: GAUGE
I dont see any metrics coming out of JMX with prefix snowflake.kafka.connector. However, I see the default metrics from Kafka connector. For example, I am getting below metrics.
kafka_connect_connector_metrics{connector_class="com.snowflake.kafka.connector.SnowflakeSinkConnector",connector="gpt-dv-eligibility-cob-v10-dev",} 1.0 kafka_connect_connector_metrics{connector_version="1.8.1",connector="gpt-dv-eligibility-cob-v10-dev",} 1.0 kafka_connect_connector_metrics{connector_type="sink",connector="gpt-dv-eligibility-cob-v10-dev",} 1.0 kafka_connect_connector_metrics{status="running",connector="gpt-dv-eligibility-cob-v10-dev",} 1.0 kafka_connect_connector_metrics{connector_class="com.snowflake.kafka.connector.SnowflakeSinkConnector",connector="gpt-dv-nwpr-raw-dev",} 1.0 kafka_connect_connector_metrics{connector_version="1.8.1",connector="gpt-dv-nwpr-raw-dev",} 1.0 kafka_connect_connector_metrics{connector_type="sink",connector="gpt-dv-nwpr-raw-dev",} 1.0 kafka_connect_connector_metrics{status="running",connector="gpt-dv-nwpr-raw-dev",} 1.0
These are the configs that i mentioned in my docker build file.
Connect JMX config
ENV JMX_PORT=9280 ENV KAFKA_JMX_HOSTNAME=127.0.0.1 ENV JMX_OPT=true
ENV KAFKA_JMX_OPTS="-javaagent:/usr/share/java/kafka-connect/jmx_prometheus_javaagent-0.17.2.jar=8085:/opt/optum/jmx-config.yaml -Djava.rmi.server.hostname=${KAFKA_JMX_HOSTNAME} -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9280 -Dcom.sun.management.jmxremote.rmi.port=9280 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
Kindly review and suggest.
lowercaseOutputName: true
rules:
# Example: snowflake.kafka.connector:connector=SnowflakeSinkConnector_SS_JP_Test,pipe=SNOWFLAKE_KAFKA_CONNECTOR_SnowflakeSinkConnector_SS_JP_Test_PIPE_JUNE_21_SS_LOAD_LOCAL_0,category=file-counts,name=file-count-on-ingestion
- pattern: 'snowflake.kafka.connector<connector=(.+), pipe=(.+), category=(.+), name=(.+)><>(.+): (.+)'
name: snowflake_kc_$3_$4_$5
value: $6
labels:
connectorName: $1
pipeName: $2
category: $3
metric: $5
type: GAUGE
Note the space after comma in pattern line.
Thanks for the response. I have tried with this pattern only in my JMX config file and i still couldnt get these metrics. Could you please help me with this issue.
This is the docker file that i am using and i am trying to listen to metrics on URL http://localhost:9403/metrics.
Still I am not getting any metrics related to snowflake. I am getting only JVM metrics.
HELP jvm_gc_collection_seconds Time spent in a given JVM garbage collector in seconds.
TYPE jvm_gc_collection_seconds summary
jvm_gc_collection_seconds_count{gc="G1 Young Generation",} 132.0 jvm_gc_collection_seconds_sum{gc="G1 Young Generation",} 2.088 jvm_gc_collection_seconds_count{gc="G1 Old Generation",} 0.0 jvm_gc_collection_seconds_sum{gc="G1 Old Generation",} 0.0
HELP jvm_memory_pool_allocated_bytes_total Total bytes allocated in a given JVM memory pool. Only updated after GC, not continuously.
TYPE jvm_memory_pool_allocated_bytes_total counter
jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'profiled nmethods'",} 3.0817408E7 jvm_memory_pool_allocated_bytes_total{pool="G1 Old Gen",} 2.4448564E8 jvm_memory_pool_allocated_bytes_total{pool="G1 Eden Space",} 1.737490432E10 jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'non-profiled nmethods'",} 1.1563136E7 jvm_memory_pool_allocated_bytes_total{pool="G1 Survivor Space",} 1.11149056E8 jvm_memory_pool_allocated_bytes_total{pool="Compressed Class Space",} 2.0968E7 jvm_memory_pool_allocated_bytes_total{pool="Metaspace",} 1.48893824E8 jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'non-nmethods'",} 1506176.0
HELP jmx_exporter_build_info A metric with a constant '1' value labeled with the version of the JMX exporter.
TYPE jmx_exporter_build_info gauge
jmx_exporter_build_info{version="0.17.2",name="jmx_prometheus_javaagent",} 1.0
HELP jmx_config_reload_failure_total Number of times configuration have failed to be reloaded.
TYPE jmx_config_reload_failure_total counter
jmx_config_reload_failure_total 0.0
HELP jvm_info VM version info
TYPE jvm_info gauge
jvm_info{runtime="OpenJDK Runtime Environment",vendor="Azul Systems, Inc.",version="11.0.16.1+1-LTS",} 1.0
HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 230.66
HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
TYPE process_start_time_seconds gauge
process_start_time_seconds 1.670582979265E9
HELP process_open_fds Number of open file descriptors.
TYPE process_open_fds gauge
process_open_fds 987.0
HELP process_max_fds Maximum number of open file descriptors.
TYPE process_max_fds gauge
process_max_fds 1048576.0
HELP process_virtual_memory_bytes Virtual memory size in bytes.
TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 9.687662592E9
HELP process_resident_memory_bytes Resident memory size in bytes.
TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 1.474605056E9
HELP jmx_scrape_duration_seconds Time this JMX scrape took, in seconds.
TYPE jmx_scrape_duration_seconds gauge
jmx_scrape_duration_seconds 0.1118809
HELP jmx_scrape_error Non-zero if this scrape failed.
TYPE jmx_scrape_error gauge
jmx_scrape_error 0.0
HELP jmx_scrape_cached_beans Number of beans with their matching rule cached
TYPE jmx_scrape_cached_beans gauge
jmx_scrape_cached_beans 0.0
HELP jvm_classes_currently_loaded The number of classes that are currently loaded in the JVM
TYPE jvm_classes_currently_loaded gauge
jvm_classes_currently_loaded 32744.0
HELP jvm_classes_loaded_total The total number of classes that have been loaded since the JVM has started execution
TYPE jvm_classes_loaded_total counter
jvm_classes_loaded_total 36965.0
HELP jvm_classes_unloaded_total The total number of classes that have been unloaded since the JVM has started execution
TYPE jvm_classes_unloaded_total counter
jvm_classes_unloaded_total 4221.0
HELP jmx_config_reload_success_total Number of times configuration have successfully been reloaded.
TYPE jmx_config_reload_success_total counter
jmx_config_reload_success_total 0.0
HELP jvm_threads_current Current thread count of a JVM
TYPE jvm_threads_current gauge
jvm_threads_current 197.0
HELP jvm_threads_daemon Daemon thread count of a JVM
TYPE jvm_threads_daemon gauge
jvm_threads_daemon 109.0
HELP jvm_threads_peak Peak thread count of a JVM
TYPE jvm_threads_peak gauge
jvm_threads_peak 198.0
HELP jvm_threads_started_total Started thread count of a JVM
TYPE jvm_threads_started_total counter
jvm_threads_started_total 298.0
HELP jvm_threads_deadlocked Cycles of JVM-threads that are in deadlock waiting to acquire object monitors or ownable synchronizers
TYPE jvm_threads_deadlocked gauge
jvm_threads_deadlocked 0.0
HELP jvm_threads_deadlocked_monitor Cycles of JVM-threads that are in deadlock waiting to acquire object monitors
TYPE jvm_threads_deadlocked_monitor gauge
jvm_threads_deadlocked_monitor 0.0
HELP jvm_threads_state Current count of threads by state
TYPE jvm_threads_state gauge
jvm_threads_state{state="NEW",} 0.0 jvm_threads_state{state="TERMINATED",} 0.0 jvm_threads_state{state="RUNNABLE",} 48.0 jvm_threads_state{state="BLOCKED",} 95.0 jvm_threads_state{state="WAITING",} 41.0 jvm_threads_state{state="TIMED_WAITING",} 13.0 jvm_threads_state{state="UNKNOWN",} 0.0
HELP jvm_memory_objects_pending_finalization The number of objects waiting in the finalizer queue.
TYPE jvm_memory_objects_pending_finalization gauge
jvm_memory_objects_pending_finalization 0.0
HELP jvm_memory_bytes_used Used bytes of a given JVM memory area.
TYPE jvm_memory_bytes_used gauge
jvm_memory_bytes_used{area="heap",} 4.059264E8 jvm_memory_bytes_used{area="nonheap",} 2.0814068E8
HELP jvm_memory_bytes_committed Committed (bytes) of a given JVM memory area.
TYPE jvm_memory_bytes_committed gauge
jvm_memory_bytes_committed{area="heap",} 2.43269632E9 jvm_memory_bytes_committed{area="nonheap",} 2.13962752E8
HELP jvm_memory_bytes_max Max (bytes) of a given JVM memory area.
TYPE jvm_memory_bytes_max gauge
jvm_memory_bytes_max{area="heap",} 6.442450944E9 jvm_memory_bytes_max{area="nonheap",} -1.0
HELP jvm_memory_bytes_init Initial bytes of a given JVM memory area.
TYPE jvm_memory_bytes_init gauge
jvm_memory_bytes_init{area="heap",} 1.073741824E9 jvm_memory_bytes_init{area="nonheap",} 7667712.0
HELP jvm_memory_pool_bytes_used Used bytes of a given JVM memory pool.
TYPE jvm_memory_pool_bytes_used gauge
jvm_memory_pool_bytes_used{pool="CodeHeap 'non-nmethods'",} 1467008.0 jvm_memory_pool_bytes_used{pool="Metaspace",} 1.45059368E8 jvm_memory_pool_bytes_used{pool="CodeHeap 'profiled nmethods'",} 3.045696E7 jvm_memory_pool_bytes_used{pool="Compressed Class Space",} 1.990704E7 jvm_memory_pool_bytes_used{pool="G1 Eden Space",} 2.08666624E8 jvm_memory_pool_bytes_used{pool="G1 Old Gen",} 1.9096832E8 jvm_memory_pool_bytes_used{pool="G1 Survivor Space",} 6291456.0 jvm_memory_pool_bytes_used{pool="CodeHeap 'non-profiled nmethods'",} 1.1250304E7
HELP jvm_memory_pool_bytes_committed Committed bytes of a given JVM memory pool.
TYPE jvm_memory_pool_bytes_committed gauge
jvm_memory_pool_bytes_committed{pool="CodeHeap 'non-nmethods'",} 2555904.0 jvm_memory_pool_bytes_committed{pool="Metaspace",} 1.47656704E8 jvm_memory_pool_bytes_committed{pool="CodeHeap 'profiled nmethods'",} 3.1522816E7 jvm_memory_pool_bytes_committed{pool="Compressed Class Space",} 2.0758528E7 jvm_memory_pool_bytes_committed{pool="G1 Eden Space",} 3.00941312E8 jvm_memory_pool_bytes_committed{pool="G1 Old Gen",} 2.125463552E9 jvm_memory_pool_bytes_committed{pool="G1 Survivor Space",} 6291456.0 jvm_memory_pool_bytes_committed{pool="CodeHeap 'non-profiled nmethods'",} 1.14688E7
HELP jvm_memory_pool_bytes_max Max bytes of a given JVM memory pool.
TYPE jvm_memory_pool_bytes_max gauge
jvm_memory_pool_bytes_max{pool="CodeHeap 'non-nmethods'",} 5828608.0 jvm_memory_pool_bytes_max{pool="Metaspace",} -1.0 jvm_memory_pool_bytes_max{pool="CodeHeap 'profiled nmethods'",} 1.22912768E8 jvm_memory_pool_bytes_max{pool="Compressed Class Space",} 1.073741824E9 jvm_memory_pool_bytes_max{pool="G1 Eden Space",} -1.0 jvm_memory_pool_bytes_max{pool="G1 Old Gen",} 6.442450944E9 jvm_memory_pool_bytes_max{pool="G1 Survivor Space",} -1.0 jvm_memory_pool_bytes_max{pool="CodeHeap 'non-profiled nmethods'",} 1.22916864E8
HELP jvm_memory_pool_bytes_init Initial bytes of a given JVM memory pool.
TYPE jvm_memory_pool_bytes_init gauge
jvm_memory_pool_bytes_init{pool="CodeHeap 'non-nmethods'",} 2555904.0 jvm_memory_pool_bytes_init{pool="Metaspace",} 0.0 jvm_memory_pool_bytes_init{pool="CodeHeap 'profiled nmethods'",} 2555904.0 jvm_memory_pool_bytes_init{pool="Compressed Class Space",} 0.0 jvm_memory_pool_bytes_init{pool="G1 Eden Space",} 5.6623104E7 jvm_memory_pool_bytes_init{pool="G1 Old Gen",} 1.01711872E9 jvm_memory_pool_bytes_init{pool="G1 Survivor Space",} 0.0 jvm_memory_pool_bytes_init{pool="CodeHeap 'non-profiled nmethods'",} 2555904.0
HELP jvm_memory_pool_collection_used_bytes Used bytes after last collection of a given JVM memory pool.
TYPE jvm_memory_pool_collection_used_bytes gauge
jvm_memory_pool_collection_used_bytes{pool="G1 Eden Space",} 0.0 jvm_memory_pool_collection_used_bytes{pool="G1 Old Gen",} 0.0 jvm_memory_pool_collection_used_bytes{pool="G1 Survivor Space",} 6291456.0
HELP jvm_memory_pool_collection_committed_bytes Committed after last collection bytes of a given JVM memory pool.
TYPE jvm_memory_pool_collection_committed_bytes gauge
jvm_memory_pool_collection_committed_bytes{pool="G1 Eden Space",} 3.00941312E8 jvm_memory_pool_collection_committed_bytes{pool="G1 Old Gen",} 0.0 jvm_memory_pool_collection_committed_bytes{pool="G1 Survivor Space",} 6291456.0
HELP jvm_memory_pool_collection_max_bytes Max bytes after last collection of a given JVM memory pool.
TYPE jvm_memory_pool_collection_max_bytes gauge
jvm_memory_pool_collection_max_bytes{pool="G1 Eden Space",} -1.0 jvm_memory_pool_collection_max_bytes{pool="G1 Old Gen",} 6.442450944E9 jvm_memory_pool_collection_max_bytes{pool="G1 Survivor Space",} -1.0
HELP jvm_memory_pool_collection_init_bytes Initial after last collection bytes of a given JVM memory pool.
TYPE jvm_memory_pool_collection_init_bytes gauge
jvm_memory_pool_collection_init_bytes{pool="G1 Eden Space",} 5.6623104E7 jvm_memory_pool_collection_init_bytes{pool="G1 Old Gen",} 1.01711872E9 jvm_memory_pool_collection_init_bytes{pool="G1 Survivor Space",} 0.0
HELP jvm_buffer_pool_used_bytes Used bytes of a given JVM buffer pool.
TYPE jvm_buffer_pool_used_bytes gauge
jvm_buffer_pool_used_bytes{pool="mapped",} 0.0 jvm_buffer_pool_used_bytes{pool="direct",} 1965074.0
HELP jvm_buffer_pool_capacity_bytes Bytes capacity of a given JVM buffer pool.
TYPE jvm_buffer_pool_capacity_bytes gauge
jvm_buffer_pool_capacity_bytes{pool="mapped",} 0.0 jvm_buffer_pool_capacity_bytes{pool="direct",} 1965074.0
HELP jvm_buffer_pool_used_buffers Used buffers of a given JVM buffer pool.
TYPE jvm_buffer_pool_used_buffers gauge
jvm_buffer_pool_used_buffers{pool="mapped",} 0.0 jvm_buffer_pool_used_buffers{pool="direct",} 131.0
HELP jmx_config_reload_failure_created Number of times configuration have failed to be reloaded.
TYPE jmx_config_reload_failure_created gauge
jmx_config_reload_failure_created 1.670582979894E9
HELP jmx_config_reload_success_created Number of times configuration have successfully been reloaded.
TYPE jmx_config_reload_success_created gauge
jmx_config_reload_success_created 1.670582979893E9
HELP jvm_memory_pool_allocated_bytes_created Total bytes allocated in a given JVM memory pool. Only updated after GC, not continuously.
TYPE jvm_memory_pool_allocated_bytes_created gauge
jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'profiled nmethods'",} 1.670582981099E9 jvm_memory_pool_allocated_bytes_created{pool="G1 Old Gen",} 1.670582981106E9 jvm_memory_pool_allocated_bytes_created{pool="G1 Eden Space",} 1.670582981107E9 jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'non-profiled nmethods'",} 1.670582981107E9 jvm_memory_pool_allocated_bytes_created{pool="G1 Survivor Space",} 1.670582981107E9 jvm_memory_pool_allocated_bytes_created{pool="Compressed Class Space",} 1.670582981107E9 jvm_memory_pool_allocated_bytes_created{pool="Metaspace",} 1.670582981107E9 jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'non-nmethods'",} 1.670582981107E9
Umm, I dont see anything wrong with your config This is what I do it in tests:
KAFKA_HEAP_OPTS="-Xms512m -Xmx10g" KAFKA_OPTS="-Djava.io.tmpdir=/tmp/jdbclogs/ -javaagent:/Users/japatel/Desktop/Kafka-Connector/prometheus/jmx_prometheus_javaagent-0.16.0.jar=8085:/Users/japatel/Desktop/Kafka-Connector/prometheus/kafka-connect-no-pattern.yml" JMX_PORT=9876 $CONFLUENT_FOLDER_NAME/bin/connect-distributed $SNOWFLAKE_APACHE_CONFIG_PATH/$SNOWFLAKE_KAFKA_CONNECT_CONFIG > $APACHE_LOG_PATH/kc.log 2>&1 &
Ignore the env variables, but you use KAFKA_JMX_OPTS and I use KAFKA_OPTS before starting kafka connect. Not sure if this would help..
Another thing you can verify:
- There is a logger page for all jmx metrics being emitted, you can search for that. It was http://localhost:8085/metrics on my local machine.
- Make sure you start kafka connector and ingest some data. it needs at least one record from kafka to start emitting metrics.
You can try this simple no pattern yml file too:
---
lowercaseOutputName: true
lowercaseOutputLabelNames: true