[GR-38414] Report statistics about gc and memory pools

[vsevel]

Feature request

Is your feature request related to a problem? Please describe. Management metrics about the gc and the memory pools are not reported through the java.lang.management.ManagementFactory GarbageCollectorMXBean and MemoryPoolMXBean mbeans.

With that code:

        log.info("found " + ManagementFactory.getGarbageCollectorMXBeans().size() + " gc");
        for (GarbageCollectorMXBean garbageCollectorMXBean : ManagementFactory.getGarbageCollectorMXBeans()) {
            log.info("==> gc " + garbageCollectorMXBean.getName()
                    + " CollectionCount=" + garbageCollectorMXBean.getCollectionCount()
                    + " CollectionTime=" + garbageCollectorMXBean.getCollectionTime()
                    + " MemoryPoolNames=" + Arrays.asList(garbageCollectorMXBean.getMemoryPoolNames()));
        }
        List<MemoryPoolMXBean> memoryPoolMXBeans = ManagementFactory.getMemoryPoolMXBeans();
        log.info("found " + memoryPoolMXBeans.size() + " memory pools");
        for (MemoryPoolMXBean memoryPoolMXBean : memoryPoolMXBeans) {
            log.info("==> memoryPoolMXBean " + memoryPoolMXBean.getName()
                    + " type=" + memoryPoolMXBean.getType()
                    + " isValid=" + memoryPoolMXBean.isValid()
                    + " getMemoryManagerNames=" + Arrays.asList(memoryPoolMXBean.getMemoryManagerNames())
                    + " usage=" + memoryPoolMXBean.getUsage()
                    + " peak usage=" + memoryPoolMXBean.getPeakUsage()
                    + " collection usage=" + memoryPoolMXBean.getCollectionUsage()
            );
        }

If I run with Serial GC in jvm mode I get:

> java -XX:+UseSerialGC -Xlog:gc=info -jar target/quarkus-app/quarkus-run.jar

found 2 gc
==> gc Copy CollectionCount=0 CollectionTime=0 MemoryPoolNames=[Eden Space, Survivor Space]
==> gc MarkSweepCompact CollectionCount=2 CollectionTime=99 MemoryPoolNames=[Eden Space, Survivor Space, Tenured Gen]
found 8 memory pools
==> memoryPoolMXBean CodeHeap 'non-nmethods' type=Non-heap memory isValid=true getMemoryManagerNames=[CodeCacheManager] usage=init = 2555904(2496K) used = 1318784(1287K) committed = 2555904(2496K) max = 5840896(5704K) peak usage=init = 2555904(2496K) used = 1318784(1287K) committed = 2555904(2496K) max = 5840896(5704K) collection usage=null
==> memoryPoolMXBean Metaspace type=Non-heap memory isValid=true getMemoryManagerNames=[Metaspace Manager] usage=init = 0(0K) used = 36996912(36129K) committed = 37486592(36608K) max = -1(-1K) peak usage=init = 0(0K) used = 36996912(36129K) committed = 37486592(36608K) max = -1(-1K) collection usage=null
==> memoryPoolMXBean Tenured Gen type=Heap memory isValid=true getMemoryManagerNames=[MarkSweepCompact] usage=init = 173408256(169344K) used = 16093464(15716K) committed = 173408256(169344K) max = 2773876736(2708864K) peak usage=init = 173408256(169344K) used = 16093464(15716K) committed = 173408256(169344K) max = 2773876736(2708864K) collection usage=init = 173408256(169344K) used = 16093464(15716K) committed = 173408256(169344K) max = 2773876736(2708864K)
==> memoryPoolMXBean CodeHeap 'profiled nmethods' type=Non-heap memory isValid=true getMemoryManagerNames=[CodeCacheManager] usage=init = 2555904(2496K) used = 6026496(5885K) committed = 6029312(5888K) max = 122908672(120028K) peak usage=init = 2555904(2496K) used = 6026496(5885K) committed = 6029312(5888K) max = 122908672(120028K) collection usage=null
==> memoryPoolMXBean Eden Space type=Heap memory isValid=true getMemoryManagerNames=[Copy, MarkSweepCompact] usage=init = 69337088(67712K) used = 7827704(7644K) committed = 69468160(67840K) max = 1109524480(1083520K) peak usage=init = 69337088(67712K) used = 65618328(64080K) committed = 69468160(67840K) max = 1109524480(1083520K) collection usage=init = 69337088(67712K) used = 0(0K) committed = 69468160(67840K) max = 1109524480(1083520K)
==> memoryPoolMXBean Survivor Space type=Heap memory isValid=true getMemoryManagerNames=[Copy, MarkSweepCompact] usage=init = 8650752(8448K) used = 0(0K) committed = 8650752(8448K) max = 138674176(135424K) peak usage=init = 8650752(8448K) used = 0(0K) committed = 8650752(8448K) max = 138674176(135424K) collection usage=init = 8650752(8448K) used = 0(0K) committed = 8650752(8448K) max = 138674176(135424K)
==> memoryPoolMXBean Compressed Class Space type=Non-heap memory isValid=true getMemoryManagerNames=[Metaspace Manager] usage=init = 0(0K) used = 4597928(4490K) committed = 4849664(4736K) max = 1073741824(1048576K) peak usage=init = 0(0K) used = 4597928(4490K) committed = 4849664(4736K) max = 1073741824(1048576K) collection usage=null
==> memoryPoolMXBean CodeHeap 'non-profiled nmethods' type=Non-heap memory isValid=true getMemoryManagerNames=[CodeCacheManager] usage=init = 2555904(2496K) used = 964480(941K) committed = 2555904(2496K) max = 122908672(120028K) peak usage=init = 2555904(2496K) used = 964480(941K) committed = 2555904(2496K) max = 122908672(120028K) collection usage=null

If I compile the application in native, I get:

found 2 gc
==> gc young generation scavenger CollectionCount=0 CollectionTime=0 MemoryPoolNames=[young generation space]
==> gc complete scavenger CollectionCount=0 CollectionTime=0 MemoryPoolNames=[young generation space, old generation space]
found 0 memory pools

If I look at prometheus as reported by micrometer, here are the metrics I am missing in GraalVM:

# HELP jvm_info_total JVM version info
# TYPE jvm_info_total counter
jvm_info_total{runtime="OpenJDK Runtime Environment",vendor="Red Hat, Inc.",version="17.0.3+6-LTS",} 1.0
# HELP jvm_buffer_memory_used_bytes An estimate of the memory that the Java virtual machine is using for this buffer pool
# TYPE jvm_buffer_memory_used_bytes gauge
jvm_buffer_memory_used_bytes{id="mapped - 'non-volatile memory'",} 0.0
jvm_buffer_memory_used_bytes{id="mapped",} 90000.0
jvm_buffer_memory_used_bytes{id="direct",} 815633.0
# HELP system_load_average_1m The sum of the number of runnable entities queued to available processors and the number of runnable entities running on the available processors averaged over a period of time
# TYPE system_load_average_1m gauge
system_load_average_1m 1.82
# HELP jvm_memory_used_bytes The amount of used memory
# TYPE jvm_memory_used_bytes gauge
jvm_memory_used_bytes{area="heap",id="Tenured Gen",} 3.2446504E7
jvm_memory_used_bytes{area="nonheap",id="CodeHeap 'profiled nmethods'",} 1.9241344E7
jvm_memory_used_bytes{area="heap",id="Eden Space",} 2.5029584E7
jvm_memory_used_bytes{area="nonheap",id="Metaspace",} 8.6850064E7
jvm_memory_used_bytes{area="nonheap",id="CodeHeap 'non-nmethods'",} 1500800.0
jvm_memory_used_bytes{area="heap",id="Survivor Space",} 32576.0
jvm_memory_used_bytes{area="nonheap",id="Compressed Class Space",} 1.0997192E7
jvm_memory_used_bytes{area="nonheap",id="CodeHeap 'non-profiled nmethods'",} 5933184.0

The only thing I get related to gc in GraalVM is:

# HELP jvm_gc_overhead_percent An approximation of the percent of CPU time used by GC activities over the last lookback period or since monitoring began, whichever is shorter, in the range [0..1]
# TYPE jvm_gc_overhead_percent gauge
jvm_gc_overhead_percent 0.0

Describe the solution you'd like. All metrics related to gc and memory pools, as we have in hotspot.

Describe who do you think will benefit the most. GraalVM users (developers and ops).

Describe alternatives you've considered. A clear and concise description of any alternative solutions or features you've considered.

Additional context. N/A

Express whether you'd like to help contributing this feature If you'd like to contribute, please read the contribution guide.

[fniephaus]

Hi @vsevel, Thanks for raising this issue. We are aware that some of these management APIs are not yet supported in GraalVM Native Image and I hope we can change this soon. Community contributions are, of course, welcome!

Tracking this internally via GR-38414.

[vsevel]

thanks @fniephaus

[vsevel]

hello. any progress on this? thanks.

[fniephaus]

Hi @vsevel, this is only one of many monitoring APIs that still needs to be supported in Native Image. Contributions are always more than welcome but at this point, I can't tell when someone in our team will have the capacity to work on this.

[vsevel]

hello any news on this?

[fniephaus]

No, I'm afraid we don't have any updates for this at the moment.

[phejl]

Any chance for https://github.com/oracle/graal/pull/5469 to be merged soon? It is crucial for production usage to have an insight at least into heap.

[sdeleuze]

FYI strong need on Spring side for this.

[vsevel]

hello any news on this?

[fniephaus]

As your example shows, we already have implementations of GarbageCollectorMXBean: CompleteGarbageCollectorMXBean and IncrementalGarbageCollectorMXBean. #5469 was opened recently with an implementation for MemoryPoolMXBean. Maybe you could take that for a spin and help us review the PR?

[fniephaus]

We've merged #5469 and the new MemoryPoolMXBean implementation will ship with the next release (GraalVM for JDK17/JDK20).

[Haarolean]

hi @fniephaus, could you please elaborate on which version should this be available in? Currently closed assigned milestone "GraalVM for JDK 17 / JDK 20 (June 13, 2023)" contains no specific version, and using the latest image (ghcr.io/graalvm/native-image:ol8-java17-22) with GraalVM 22.3.3 Java 17 CE I can see that this feature is still not there.

[fniephaus]

Hey @Haarolean, sorry for the late response. This is not available in 22.3.3. I guess you could use something like ghcr.io/graalvm/native-image-community:17.0.8-ol9 instead now.