java.lang.OutOfMemoryError: GC overhead limit exceeded

[zamani84]

Hallo, we use Zorka in JBoss EAP server and have a OutOfMemory Problem. After sending the heap dump to JBoss support they analyzed it and told us, that Zorka is making the trouble:

Class Name                                    | Objects | Shallow Heap |    Retained Heap
------------------------------------------------------------------------------------------
com.jitlogic.zorka.common.util.TapOutputStream|      93 |        3,720 | >= 1,578,233,296
org.apache.cxf.io.CacheAndWriteOutputStream   |      93 |        8,184 | >= 1,578,134,888
------------------------------------------------------------------------------------------

The heap dump shows the largest point of heap growth is 93 om.jitlogic.zorka.common.util.TapOutputStream and their underlying org.apache.cxf.io.CacheAndWriteOutputStreams. These are stored as thread locals on request threads through the _tap_xml_out com.jitlogic.zorka.deps.bsh.Variable's thread local value. Removing those thread locals would avoid substantial heap growth. Can you tell us why these streams are cached like this?

JBoss support told us, we should ask you how to drop TapOutputStream's thread local to avoid that heap overhead, because Zorka is creating thread locals that reference the cxf stream and prevent it from being GC'd. Typically, the cxf stream will not be cached on a thread local like this so it does not cause any persistent heap growth. Is Zorka specifically caching those for some kind of monitoring purpose? Can its caching of these cxf streams be disabled? If so, that change alone should drop in our case heap usage by about 1.5 gb in the provided dump.

We use Zorka 1.0.16

[zamani84]

Please can someone fix this problem? Without the zorka library the problem will not appear - I have tested it

[zamani84]

hallo somebody there?

[zamani84]

I know, that this problem happens very very rare, but somehow zorka is holding under certain conditions references. This problem happens when using very big wsdl files with a lot of other linked schema definitions under certain unkown conditions.

[jitlogic]

Yes, stream tapping is risky and possibly I should not enable this by default. Please add the following to zorka.properties:

soap.fetch.xml = no

Sorry for delay.

[jitlogic]

I'm looking for ways to handle this as I've encountered similar problems with Apache Camel. I'll let you know when I find solution to this.

[fisp-kks]

Hi, since @zamani84 is no longer involved with this issue, I wanted to give you an update on our situation. First of all thanks for the reply. We added 'soap.fetch.xml = no' to our zorka.properties. Unfortunately it didn't help and we ran into the same Memory Problem we had before. Would it be helpful for you to see our complete zorka.properties file?