Memory leak while deploying spring-cloud-netflix application on standalone tomcat

[ghost]

Tomcat fail to stop thread named ServoMonitorGetValueLimiter. A thread called ServoMonitorGetValueLimiter started by the the application which is never stopped. This thread is started by MonitorRegistryMetricPoller of com.netflix.servo . In fact service is failed to stop the thread. The object of this class is a component in ServoMetricCollector of org.springframework.cloud.netflix.servo to exposes servo metrics to the metric endpoint.

when application is undeployed it is unable to unregister the MBean also (com.netflix.servo)

[spencergibb]

Sorry for the delay in responding. Is this still an issue? Is it a memory leak, or just the thread isn't shut down properly?

[AnilK11]

I am facing similar issue. While undeploying application from tomcat it is failed to stop thread named ServoMonitorGetValueLimiter causing memory leak. After undeploying the application multiple times, a number of ServoMonitorGetValueLimiter instances remains alive in perm area which is causing java.lang.OutOfMemory exception. I am using Angel.SR4 version of spring-cloud-starter-parent.

[cforce]

Is it a tomcat only problem? What about undertow?

[dsyer]

What version of Tomcat is that?

[AnilK11]

I am using tomcat 8.0.

[dsyer]

There's an option in Tomcat to forcibly stop threads and clean up class loaders. I thought it was on by default, but maybe not (or maybe you switched it off). Does that help?

[sprgn]

This is happening in the netflix servo code spinning raw Thread MonitorRegistryMetricPoller.java 77e6036

[sprgn]

Following context attributes take care of most of the loose threads out there except for ServoMonitorGetValueLimiter.

context.xml

<?xml version="1.0" encoding="UTF-8"?>
<Context clearReferencesStopTimerThreads="true" clearReferencesStopThreads="true">
</Context>

[spencergibb]

@sprgn Are you using atlas or specatator? I don't see MonitorRegistryMetricPoller get instantiated.

[sprgn]

No, neither one. Going off the name of the thread in the stack trace, and MonitorRegistryMetricPoller is the only one that creates threads with name ServoMonitorGetValueLimiter-%d

19-Jan-2016 14:15:22.673 WARNING [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ws_eureka] appear
s to have started a thread named [ServoMonitorGetValueLimiter-0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
 java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
 java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 java.lang.Thread.run(Thread.java:745)

[sprgn]

Well, I should say I'm not using either one that I know of, at least not explicitly.

[spencergibb]

@sprgn MonitorRegistryMetricPoller doesn't get instantiated in a basic sample. Do you have a sample app that shows the problem?

[sprgn]

@spencergibb here it is link to ws_eureka app codebase

[sprgn]

It expects tomcat to be running on port 80.

[spencergibb]

Ah, I am running Brixton, you are running Angel.

[sprgn]

Running Brixton.M4 it gives similar error, but for a different named thread now

19-Jan-2016 15:17:12.879 WARNING [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ws_eureka] appear
s to have started a thread named [spring.cloud.inetutils] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
 java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
 java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 java.lang.Thread.run(Thread.java:745)

[spencergibb]

That makes sense!

[sprgn]

How can I cancel that thread?

[spencergibb]

You'd likely need to use reflection in M4. This commit should fix it.

[sprgn]

Thanks! Reflection worked to fix it for now until the fix is released. Worked via a ServletContextListener contextDestroyed method

import ch.qos.logback.classic.LoggerContext;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.cloud.util.InetUtils;
import org.springframework.stereotype.Component;

import javax.servlet.ServletContextEvent;
import javax.servlet.ServletContextListener;
import javax.servlet.annotation.WebListener;
import java.lang.reflect.Field;
import java.util.concurrent.ExecutorService;

@WebListener
@Component
public class LogbackShutdownListener implements ServletContextListener {
    private final Logger logger = LoggerFactory.getLogger(this.getClass());

    public void contextDestroyed(ServletContextEvent sce) {
        Field field = null;
        try {
            field = InetUtils.class.getDeclaredField("executor");

            field.setAccessible(true);
            Object es = field.get(null);
            ((ExecutorService)es).shutdownNow();
        } catch (Exception e) {
            //
        }

        LoggerContext context = (LoggerContext)LoggerFactory.getILoggerFactory();
        this.logger.warn("Shutting down loggerContext in contextDestroyed");
        context.stop();
    }

    public void contextInitialized(ServletContextEvent sce) {
    }
}

[sprgn]

Is Angel.SR4 not recommended to be used then at this point?

[dsyer]

Angel does not have that thread pool in it (the one referenced by Spencer as fixed in master). Are we talking at cross purposes here?

[spencergibb]

Angel has different threads that remain open. @sprgn Angel.SR4 is the GA release we support.

[jsaraiy]

@sprgn Any update on this issue ? We are getting the same error in Greenwich.M3 cloud version as well. By using shutdown hook that you mentioned before, we are able to fix the error. Wondering, if any of the future spring-cloud release will fix this.

[dsyer]

It’s probably not possible for your current issue to be the same as this one (the code changed a lot since Angel). Please open a new issue and provide detailed steps to reproduce.