Out of Java Heap space - Recommended Fix

[GasparYYC]

Hi, Sharing with the team how to fix the Java heap space error.

One of my apps just started failing (no code/configuration changes whatsover). The only change has been more data..

The user reported the following error from the web page: 500 Internal Server Error An unexpected error with the following message occurred: Java heap space

In my case in the file /usr/share/tomcat/logs/catalina.2016-08-29.log showed errors like this:

Aug 29, 2016 4:57:56 PM oracle.dbtools.http.errors.ErrorPageFilter log SEVERE: Java heap space java.lang.OutOfMemoryError: Java heap space at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:165) at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:48) at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:938) at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1150) at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:4798) at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:4901) at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:5631) at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1385) at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at oracle.ucp.jdbc.proxy.StatementProxyFactory.invoke(StatementProxyFactory.java:353) at oracle.ucp.jdbc.proxy.PreparedStatementProxyFactory.invoke(PreparedStatementProxyFactory.java:178) at oracle.ucp.jdbc.proxy.CallableStatementProxyFactory.invoke(CallableStatementProxyFactory.java:136) at com.sun.proxy.$Proxy35.execute(Unknown Source) at oracle.dbtools.apex.OWA.execute(OWA.java:152) at oracle.dbtools.apex.ModApex.handleRequest(ModApex.java:254) at oracle.dbtools.apex.ModApex.doPost(ModApex.java:180) at oracle.dbtools.apex.ModApex.service(ModApex.java:90) at oracle.dbtools.http.entrypoint.Dispatcher.dispatch(Dispatcher.java:106) at oracle.dbtools.http.entrypoint.EntryPoint$FilteredServlet.service(EntryPoint.java:123) at oracle.dbtools.http.filters.FilterChainImpl.doFilter(FilterChainImpl.java:73) at oracle.dbtools.http.forwarding.QueryFilteringRewrite.doFilter(QueryFilteringRewrite.java:89) at oracle.dbtools.http.filters.HttpFilter.doFilter(HttpFilter.java:45) at oracle.dbtools.http.filters.FilterChainImpl.doFilter(FilterChainImpl.java:64) at oracle.dbtools.http.forwarding.ForwardingFilter.doFilter(ForwardingFilter.java:59) at oracle.dbtools.http.filters.HttpFilter.doFilter(HttpFilter.java:45) at oracle.dbtools.http.filters.FilterChainImpl.doFilter(FilterChainImpl.java:64) at oracle.dbtools.http.cors.CORSPreflightFilter.doFilter(CORSPreflightFilter.java:67) at oracle.dbtools.http.filters.HttpFilter.doFilter(HttpFilter.java:45) at oracle.dbtools.http.filters.FilterChainImpl.doFilter(FilterChainImpl.java:64) at oracle.dbtools.http.cookies.auth.CookieSessionCSRFFilter.doFilter(CookieSessionCSRFFilter.java:61)

After some searching I found some inconsistent,obsolete and contradicting information on what file to edit to reconfigure the Java Heap.. I managed to figure out where to change it..

In the file: /usr/share/tomcat/conf/tomcat.conf

Add the following line to configure 312MB of Memory to the Java Heap: JAVA_OPTS="-Xmx312m"

Note: I probably shrinked the java heap based on the memory usage before and after because after the change I actually have more memory available in the VM!

free -m total used free shared buff/cache available Mem: 993 468 71 246 452 144 Swap: 4095 153 3942

After: free -m total used free shared buff/cache available Mem: 993 515 131 132 346 206 Swap: 4095 230 3865

(Note the 144 of before vs. 206 of after)

My theory: My 1GB VM ran out of space when the heap tried to grow beyond the OS capabilities.. by defining a smaller Max Limit of the JVM heap I forced java to do a more thorough garbage collection hence behaving better..

So there you go.. just sharing..

I think it would be a good idea to add a "Troubleshooting and Best Practices" section for OXAR installations that could have information like this.

Regards

[martindsouza]

@GasparYYC thanks for the tip. Why the 312m though? I.e. where is that number coming from?

[GasparYYC]

Hi, The value of 312 was somewhat arbitrary because initially Java was reporting 250 MB of MaxHeapSize used (262144000 is in bytes):

/usr/bin/java -XX:+PrintFlagsFinal -version |grep HeapSize uintx ErgoHeapSizeLimit = 0 {product} uintx HeapSizePerGCThread = 87241520 {product} uintx InitialHeapSize := 16777216 {product} uintx LargePageHeapSizeThreshold = 134217728 {product} uintx MaxHeapSize := 262144000 {product} openjdk version "1.8.0_91" OpenJDK Runtime Environment (build 1.8.0_91-b14) OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)

So initially I thought I was increasing it.. but turns out that the reporting of memory using the command above is possibly showing incorrect results for OpenJDK. When I checked the memory in the VM I saw that I actually reduced it. The new value of 312MB on a VM with 1GB of RAM with OXAR seems to be working fine and is as fast as before.

If I had to recommend an initial value I would definitely go with 312MB. Seems to be a sweet spot.

-G