JMX_Exporter not responding

[uberspot]

trafficstars

I seem to be getting a similar issue as: https://github.com/prometheus/jmx_exporter/issues/487

jmx_exporter version: 0.16.1

ELK config: -javaagent:/opt/jmxexporter/jmx_prometheus_javaagent-0.16.1.jar=7071:/opt/jmxexporter/elasticsearch-jmx.yml

elasticsearch-jmx.xml:

lowercaseOutputName: true

Once I restart ELK and try to curl -v localhost:7071/ it works the first 1-2 times (I get legitimate metrics) but then it stops and I only get "Connection reset" errors which indicates the jar crashed (?).

Any way to debug this further?

(please don't repoint me to the prometheus mailing list, I believe this looks like an exporter issue and not related to prometheus directly)

Many thanks in advance! 👍

[fstab]

Here's an idea how to debug this: You could try to figure out whether this is caused by a specific JMX bean.

Use the whitelistObjectNames configuration so that jmx_exporter just scrapes a single bean. If that works, add more names to whitelistObjectNames until you learn which bean causes this.

[dhoard]

@uberspot You can also try to run the JMXScraper code manually https://github.com/prometheus/jmx_exporter/blob/5c2c9216622342116b0068d4cc47f71a37093d3a/collector/src/main/java/io/prometheus/jmx/JmxScraper.java#L334 to try to see if a specific MBean is hanging the collection.

[ebuildy]

Using an empty list give the same error

config.xml

whitelistObjectNames: []
rules:
- pattern: ".*"

~$ curl localhost:8080/metrics -v

*   Trying 127.0.0.1:8080...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET /metrics HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

[fstab]

This is weird, an empty whitelistObjectNames should result in an empty list of metrics, so I would expect that the response is empty but there should not be an error.

Please post how exactly you start the application and jmx_exporter.

[tuhoanganh]

This is weird, an empty whitelistObjectNames should result in an empty list of metrics, so I would expect that the response is empty but there should not be an error.

Please post how exactly you start the application and jmx_exporter.

Hi @fstab, i meet the same problem with @uberspot. Below is my current version and configurations:

jmx_exporter VERSION:

jmx_prometheus_javaagent-0.17.0.jar

Elasticsearch VERSION:

curl -XGET -k -u elastic:abcd1234 https://localhost:9200
{
  "name" : "ES-01",
  "cluster_name" : "aaa",
  "cluster_uuid" : "vmPHQfEqQ0iODJ4-EvUevQ",
  "version" : {
    "number" : "8.2.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "b174af62e8dd9f4ac4d25875e9381ffe2b9282c5",
    "build_date" : "2022-04-20T10:35:10.180408517Z",
    "build_snapshot" : false,
    "lucene_version" : "9.1.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

Add jmx_exporter to Elasticsearch:

[root@ES-01 ~]# cat /etc/elasticsearch/jvm.options.d/test.options 
-javaagent:/opt/jmx_exporter/jmx_prometheus_javaagent-0.17.0.jar=8088:/opt/jmx_exporter/java_process.yml

java_process.yml's content:

[root@ES-01 ~]# cat /opt/jmx_exporter/java_process.yml 
---
startDelaySeconds: 0
lowercaseOutputLabelNames: false
ssl: false
lowercaseOutputName: false
whitelistObjectNames:
  - "java.lang:type=OperatingSystem"

JMX port after restart Elasticsearch service:

[root@ES-01 ~]# netstat -anp |grep 8088
tcp6       0      0 :::8088                 :::*                    LISTEN      10782/java      

Testing with CURL:

[root@ES-01 ~]#  curl http://127.0.0.1:8088/metrics  -v
* About to connect() to 127.0.0.1 port 8088 (#0)
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 8088 (#0)
> GET /metrics HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 127.0.0.1:8088
> Accept: */*
> 
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

As @uberspot mentioned above, i only successfully curl the metrics in the very beginning of starting elasticsearch process.

Once I restart ELK and try to curl -v localhost:7071/ it works the first 1-2 times (I get legitimate metrics) but then it stops and I only get "Connection reset" errors which indicates the jar crashed (?).

[fstab]

Thanks for the detailed report. As you have whitelisted java.lang:type=OperatingSystem it would be good to know:

• What operating system are you using? Maybe there's something unexpected in your OS metrics. Let me know what your system is, then I'll try to reproduce it.
• Does this have anything to do with Elastic, or does this happen with any Java application on that system? The integration tests in jmx_exporter come with a simple example application. If you build jmx_exporter from source with ./mvnw package the example application will be created in ./integration_tests/jmx_example_application/target/jmx_example_application.jar. It would be great if you could run the example application instead of Elastic with the Java agent attached to see if you get the same behavior:

  java -javaagent:/opt/jmx_exporter/jmx_prometheus_javaagent-0.17.0.jar=8088:/opt/jmx_exporter/java_process.yml -jar jmx_example_application.jar
  

[tuhoanganh]

Hi @fstab, Thank you for the fast reply and support.

Below are the more information in my case.

• What operating system are you using? Maybe there's something unexpected in your OS metrics. Let me know what your system is, then I'll try to reproduce it.

My operating system is CentOS 7:

[root@ES-01 ~]#  python -m platform
Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-centos-7.9.2009-Core

• Does this have anything to do with Elastic, or does this happen with any Java application on that system? The integration tests in jmx_exporter come with a simple example application...

Yesterday, I try jmx_exporter with logstash in the same above Server and jmx_exporter working as expected for logstash.

Example Logtash's Metrics Result. [Click to expand]

# HELP jvm_classes_loaded The number of classes that are currently loaded in the JVM
# TYPE jvm_classes_loaded gauge
jvm_classes_loaded 16710.0
# HELP jvm_classes_loaded_total The total number of classes that have been loaded since the JVM has started execution
# TYPE jvm_classes_loaded_total counter
jvm_classes_loaded_total 16713.0
# HELP jvm_classes_unloaded_total The total number of classes that have been unloaded since the JVM has started execution
# TYPE jvm_classes_unloaded_total counter
jvm_classes_unloaded_total 3.0
# HELP jvm_threads_current Current thread count of a JVM
# TYPE jvm_threads_current gauge
jvm_threads_current 15.0
# HELP jvm_threads_daemon Daemon thread count of a JVM
# TYPE jvm_threads_daemon gauge
jvm_threads_daemon 14.0
# HELP jvm_threads_peak Peak thread count of a JVM
# TYPE jvm_threads_peak gauge
jvm_threads_peak 15.0
# HELP jvm_threads_started_total Started thread count of a JVM
# TYPE jvm_threads_started_total counter
jvm_threads_started_total 19.0
# HELP jvm_threads_deadlocked Cycles of JVM-threads that are in deadlock waiting to acquire object monitors or ownable synchronizers
# TYPE jvm_threads_deadlocked gauge
jvm_threads_deadlocked 0.0
# HELP jvm_threads_deadlocked_monitor Cycles of JVM-threads that are in deadlock waiting to acquire object monitors
# TYPE jvm_threads_deadlocked_monitor gauge
jvm_threads_deadlocked_monitor 0.0
# HELP jvm_threads_state Current count of threads by state
# TYPE jvm_threads_state gauge
jvm_threads_state{state="BLOCKED",} 0.0
jvm_threads_state{state="RUNNABLE",} 6.0
jvm_threads_state{state="WAITING",} 3.0
jvm_threads_state{state="TIMED_WAITING",} 6.0
jvm_threads_state{state="NEW",} 0.0
jvm_threads_state{state="TERMINATED",} 0.0
# HELP jvm_memory_bytes_used Used bytes of a given JVM memory area.
# TYPE jvm_memory_bytes_used gauge
jvm_memory_bytes_used{area="heap",} 4.04525216E8
jvm_memory_bytes_used{area="nonheap",} 1.43582968E8
# HELP jvm_memory_bytes_committed Committed (bytes) of a given JVM memory area.
# TYPE jvm_memory_bytes_committed gauge
jvm_memory_bytes_committed{area="heap",} 1.037959168E9
jvm_memory_bytes_committed{area="nonheap",} 1.62029568E8
# HELP jvm_memory_bytes_max Max (bytes) of a given JVM memory area.
# TYPE jvm_memory_bytes_max gauge
jvm_memory_bytes_max{area="heap",} 1.037959168E9
jvm_memory_bytes_max{area="nonheap",} -1.0
# HELP jvm_memory_bytes_init Initial bytes of a given JVM memory area.
# TYPE jvm_memory_bytes_init gauge
jvm_memory_bytes_init{area="heap",} 1.073741824E9
jvm_memory_bytes_init{area="nonheap",} 7667712.0
# HELP jvm_memory_pool_bytes_used Used bytes of a given JVM memory pool.
# TYPE jvm_memory_pool_bytes_used gauge
jvm_memory_pool_bytes_used{pool="CodeHeap 'non-nmethods'",} 1485952.0
jvm_memory_pool_bytes_used{pool="Metaspace",} 8.647292E7
jvm_memory_pool_bytes_used{pool="CodeHeap 'profiled nmethods'",} 3.1615104E7
jvm_memory_pool_bytes_used{pool="Compressed Class Space",} 1.3205024E7
jvm_memory_pool_bytes_used{pool="Par Eden Space",} 2.8480284E8
jvm_memory_pool_bytes_used{pool="Par Survivor Space",} 3.5782656E7
jvm_memory_pool_bytes_used{pool="CodeHeap 'non-profiled nmethods'",} 1.0803968E7
jvm_memory_pool_bytes_used{pool="CMS Old Gen",} 8.393972E7
# HELP jvm_memory_pool_bytes_committed Committed bytes of a given JVM memory pool.
# TYPE jvm_memory_pool_bytes_committed gauge
jvm_memory_pool_bytes_committed{pool="CodeHeap 'non-nmethods'",} 2555904.0
jvm_memory_pool_bytes_committed{pool="Metaspace",} 9.8033664E7
jvm_memory_pool_bytes_committed{pool="CodeHeap 'profiled nmethods'",} 3.3619968E7
jvm_memory_pool_bytes_committed{pool="Compressed Class Space",} 1.7006592E7
jvm_memory_pool_bytes_committed{pool="Par Eden Space",} 2.86326784E8
jvm_memory_pool_bytes_committed{pool="Par Survivor Space",} 3.5782656E7
jvm_memory_pool_bytes_committed{pool="CodeHeap 'non-profiled nmethods'",} 1.081344E7
jvm_memory_pool_bytes_committed{pool="CMS Old Gen",} 7.15849728E8
# HELP jvm_memory_pool_bytes_max Max bytes of a given JVM memory pool.
# TYPE jvm_memory_pool_bytes_max gauge
jvm_memory_pool_bytes_max{pool="CodeHeap 'non-nmethods'",} 5836800.0
jvm_memory_pool_bytes_max{pool="Metaspace",} -1.0
jvm_memory_pool_bytes_max{pool="CodeHeap 'profiled nmethods'",} 1.22908672E8
jvm_memory_pool_bytes_max{pool="Compressed Class Space",} 1.073741824E9
jvm_memory_pool_bytes_max{pool="Par Eden Space",} 2.86326784E8
jvm_memory_pool_bytes_max{pool="Par Survivor Space",} 3.5782656E7
jvm_memory_pool_bytes_max{pool="CodeHeap 'non-profiled nmethods'",} 1.22912768E8
jvm_memory_pool_bytes_max{pool="CMS Old Gen",} 7.15849728E8
# HELP jvm_memory_pool_bytes_init Initial bytes of a given JVM memory pool.
# TYPE jvm_memory_pool_bytes_init gauge
jvm_memory_pool_bytes_init{pool="CodeHeap 'non-nmethods'",} 2555904.0
jvm_memory_pool_bytes_init{pool="Metaspace",} 0.0
jvm_memory_pool_bytes_init{pool="CodeHeap 'profiled nmethods'",} 2555904.0
jvm_memory_pool_bytes_init{pool="Compressed Class Space",} 0.0
jvm_memory_pool_bytes_init{pool="Par Eden Space",} 2.86326784E8
jvm_memory_pool_bytes_init{pool="Par Survivor Space",} 3.5782656E7
jvm_memory_pool_bytes_init{pool="CodeHeap 'non-profiled nmethods'",} 2555904.0
jvm_memory_pool_bytes_init{pool="CMS Old Gen",} 7.15849728E8
# HELP jvm_gc_collection_seconds Time spent in a given JVM garbage collector in seconds.
# TYPE jvm_gc_collection_seconds summary
jvm_gc_collection_seconds_count{gc="ParNew",} 5.0
jvm_gc_collection_seconds_sum{gc="ParNew",} 0.768
jvm_gc_collection_seconds_count{gc="ConcurrentMarkSweep",} 2.0
jvm_gc_collection_seconds_sum{gc="ConcurrentMarkSweep",} 0.295
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 304.98
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.653555769778E9
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 90.0
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 16384.0
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 5.123244032E9
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 7.793664E8
# HELP jmx_exporter_build_info A metric with a constant '1' value labeled with the version of the JMX exporter.
# TYPE jmx_exporter_build_info gauge
jmx_exporter_build_info{version="0.13.0",name="jmx_prometheus_javaagent",} 1.0
# HELP jmx_config_reload_failure_total Number of times configuration have failed to be reloaded.
# TYPE jmx_config_reload_failure_total counter
jmx_config_reload_failure_total 0.0
# HELP jvm_buffer_pool_used_bytes Used bytes of a given JVM buffer pool.
# TYPE jvm_buffer_pool_used_bytes gauge
jvm_buffer_pool_used_bytes{pool="mapped",} 0.0
jvm_buffer_pool_used_bytes{pool="direct",} 16384.0
# HELP jvm_buffer_pool_capacity_bytes Bytes capacity of a given JVM buffer pool.
# TYPE jvm_buffer_pool_capacity_bytes gauge
jvm_buffer_pool_capacity_bytes{pool="mapped",} 0.0
jvm_buffer_pool_capacity_bytes{pool="direct",} 16384.0
# HELP jvm_buffer_pool_used_buffers Used buffers of a given JVM buffer pool.
# TYPE jvm_buffer_pool_used_buffers gauge
jvm_buffer_pool_used_buffers{pool="mapped",} 0.0
jvm_buffer_pool_used_buffers{pool="direct",} 2.0
# HELP jvm_memory_pool_allocated_bytes_total Total bytes allocated in a given JVM memory pool. Only updated after GC, not continuously.
# TYPE jvm_memory_pool_allocated_bytes_total counter
jvm_memory_pool_allocated_bytes_total{pool="Par Survivor Space",} 3.6642536E7
jvm_memory_pool_allocated_bytes_total{pool="CMS Old Gen",} 1.22019184E8
jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'profiled nmethods'",} 3.3343232E7
jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'non-profiled nmethods'",} 1.1058816E7
jvm_memory_pool_allocated_bytes_total{pool="Compressed Class Space",} 1.352748E7
jvm_memory_pool_allocated_bytes_total{pool="Metaspace",} 8.794244E7
jvm_memory_pool_allocated_bytes_total{pool="Par Eden Space",} 1.586821416E9
jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'non-nmethods'",} 1489664.0
# HELP jmx_config_reload_success_total Number of times configuration have successfully been reloaded.
# TYPE jmx_config_reload_success_total counter
jmx_config_reload_success_total 0.0
# HELP jvm_info JVM version info
# TYPE jvm_info gauge
jvm_info{version="11.0.14.1+1",vendor="Eclipse Adoptium",runtime="OpenJDK Runtime Environment",} 1.0
# HELP java_lang_OperatingSystem_FreePhysicalMemorySize FreePhysicalMemorySize (java.lang<type=OperatingSystem><>FreePhysicalMemorySize)
# TYPE java_lang_OperatingSystem_FreePhysicalMemorySize untyped
java_lang_OperatingSystem_FreePhysicalMemorySize 1.50863872E8
# HELP java_lang_OperatingSystem_ProcessCpuLoad ProcessCpuLoad (java.lang<type=OperatingSystem><>ProcessCpuLoad)
# TYPE java_lang_OperatingSystem_ProcessCpuLoad untyped
java_lang_OperatingSystem_ProcessCpuLoad 3.4525655980540304E-4
# HELP java_lang_OperatingSystem_MaxFileDescriptorCount MaxFileDescriptorCount (java.lang<type=OperatingSystem><>MaxFileDescriptorCount)
# TYPE java_lang_OperatingSystem_MaxFileDescriptorCount untyped
java_lang_OperatingSystem_MaxFileDescriptorCount 16384.0
# HELP java_lang_OperatingSystem_TotalPhysicalMemorySize TotalPhysicalMemorySize (java.lang<type=OperatingSystem><>TotalPhysicalMemorySize)
# TYPE java_lang_OperatingSystem_TotalPhysicalMemorySize untyped
java_lang_OperatingSystem_TotalPhysicalMemorySize 1.0314469376E10
# HELP java_lang_OperatingSystem_SystemCpuLoad SystemCpuLoad (java.lang<type=OperatingSystem><>SystemCpuLoad)
# TYPE java_lang_OperatingSystem_SystemCpuLoad untyped
java_lang_OperatingSystem_SystemCpuLoad 0.01830419640039065
# HELP java_lang_OperatingSystem_CommittedVirtualMemorySize CommittedVirtualMemorySize (java.lang<type=OperatingSystem><>CommittedVirtualMemorySize)
# TYPE java_lang_OperatingSystem_CommittedVirtualMemorySize untyped
java_lang_OperatingSystem_CommittedVirtualMemorySize 5.12534528E9
# HELP java_lang_OperatingSystem_AvailableProcessors AvailableProcessors (java.lang<type=OperatingSystem><>AvailableProcessors)
# TYPE java_lang_OperatingSystem_AvailableProcessors untyped
java_lang_OperatingSystem_AvailableProcessors 8.0
# HELP java_lang_OperatingSystem_TotalSwapSpaceSize TotalSwapSpaceSize (java.lang<type=OperatingSystem><>TotalSwapSpaceSize)
# TYPE java_lang_OperatingSystem_TotalSwapSpaceSize untyped
java_lang_OperatingSystem_TotalSwapSpaceSize 4.160745472E9
# HELP java_lang_OperatingSystem_FreeSwapSpaceSize FreeSwapSpaceSize (java.lang<type=OperatingSystem><>FreeSwapSpaceSize)
# TYPE java_lang_OperatingSystem_FreeSwapSpaceSize untyped
java_lang_OperatingSystem_FreeSwapSpaceSize 4.159164416E9
# HELP java_lang_OperatingSystem_OpenFileDescriptorCount OpenFileDescriptorCount (java.lang<type=OperatingSystem><>OpenFileDescriptorCount)
# TYPE java_lang_OperatingSystem_OpenFileDescriptorCount untyped
java_lang_OperatingSystem_OpenFileDescriptorCount 90.0
# HELP java_lang_OperatingSystem_SystemLoadAverage SystemLoadAverage (java.lang<type=OperatingSystem><>SystemLoadAverage)
# TYPE java_lang_OperatingSystem_SystemLoadAverage untyped
java_lang_OperatingSystem_SystemLoadAverage 0.09
# HELP java_lang_OperatingSystem_ProcessCpuTime ProcessCpuTime (java.lang<type=OperatingSystem><>ProcessCpuTime)
# TYPE java_lang_OperatingSystem_ProcessCpuTime untyped
java_lang_OperatingSystem_ProcessCpuTime 3.0503E11
# HELP jmx_scrape_duration_seconds Time this JMX scrape took, in seconds.
# TYPE jmx_scrape_duration_seconds gauge
jmx_scrape_duration_seconds 0.01533153
# HELP jmx_scrape_error Non-zero if this scrape failed.
# TYPE jmx_scrape_error gauge
jmx_scrape_error 0.0

So i think the conflict in this situation is just between Elasticsearch and jmx_exporter

[dhoard]

@uberspot @ebuildy @tuhoanganh

Have you resolved this issue?

If there are no updates within 1 week, this will be closed as inactive.