Clemens Utschig

https://support.cloudbees.com/hc/en-us/articles/360029574172-Metaspace-Leak-Due-to-classes-not-being-cleaned-up but unfortunately - we are using [Jenkins 2.289.1] :(

``` Class (reserved=818996KB +191857KB, committed=352564KB +213617KB) (classes #42887 +21410) ( instance classes #41126 +20650, array classes #1761 +760) (malloc=12084KB +7537KB #243483 +135258) (mmap: reserved=806912KB +184320KB, committed=340480KB +206080KB) ( Metadata: )...

ok - some more testing with classloader stats after start (NO running jobs): ``` Total = 897 21312 135653376 123512712 ChunkSz: Total size of all allocated metaspace chunks BlockSz: Total...

I started to play with the heap dump - and classloaders .... and all big classloaders have the common ancestor:  if I look into the classloader - this is...

The annoying thing is that in a second threaddump hours later these guys are gone ;-( so it must be something else .. https://issues.jenkins.io/plugins/servlet/mobile#issue/JENKINS-50223

Ok - findings: a) 4g min/max request memory on container level do it - with the following settings b) `-server -XX:NativeMemoryTracking=summary -XX:MaxRAMPercentage=90 -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent -XX:+ParallelRefProcEnabled -XX:+UseStringDeduplication -XX:MaxMetaspaceSize=1g -XX:MetaspaceSize=256M` and c)...

still there is the leak - albeit less .. (we have 40+ of the below) - which in fact could be one per run  per run ....  working...

to go here: https://github.com/opendevstack/ods-core/blob/master/jenkins/master/ods-run.sh#L96 ``` sh-4.2$ cat loadgrapes.sh #!/bin/sh echo $HOME ivyhome=/tmp/ivy unzip -o $HOME/plugins/workflow-cps-global-lib.jpi -d $ivyhome java -cp $HOME/war/WEB-INF/lib/groovy-all-2.4.12.jar:$ivyhome/WEB-INF/lib/ivy-2.4.0.jar -Dgrape.config=$HOME/.groovy/grapeConfig.xml -Duser.home=$HOME groovy.ui.GroovyMain loadgrapes.groovy rm -rf $ivyhome sh-4.2$ cat loadgrapes.groovy...

ok - update ... I think we have found the leak ... an assumption to be proven we see circular references from Classes loaded via GroovyCleanClassloader (=> the RUN classloader)...

https://github.com/opendevstack/ods-jenkins-shared-library/blob/master/src/org/ods/orchestration/service/JiraService.groovy#240 - my proposal - if empty/null - just warn and set to `N/A - please check the test logs` - that way it's really super surgical