GeoApiContext should clean up working threads on exit.

[pooi-pooi]

Hello guys, I had a problem while doing a PlacesApi.placeAutocomplete request I really don't know if that is a issue caused by the API or on our implementation.

We are developing an application that uses:

• Spring Boot 1.3.6.RELEASE
• Undertow
• com.google.maps:google-maps-services:0.1.17

# application.properties
server.undertow.worker-threads=1000
server.undertow.io-threads=100

# EC2 AWS
m4.xlarge

# Dockerfile

FROM java:8
RUN apt-get update && apt-get install -y

WORKDIR /app
VOLUME /root/.gradle

COPY gradle/ /app/gradle/
COPY gradlew /app/gradlew
COPY build.gradle /app/build.gradle
COPY src/ /app/src/

COPY docker-entrypoint.sh /
ENTRYPOINT ["/docker-entrypoint.sh"]

CMD ["java","-jar","/app/build/libs/application-1.0.0.jar"]

# docker-entrypoint.sh

#!/bin/bash
set -e
./gradlew build -x test
exec "$@"

@Service
public class GoogleMapsAPIService {
    public List placeAutoComplete(String input, Double latitude, Double longitude, String language) {
        GeoApiContext context = new GeoApiContext().setApiKey(GOOGLEMAPS_KEY);
        LatLng location = new LatLng(latitude, longitude); // -23.52488881961586,-46.67578987777233

        AutocompletePrediction[] autocompletePredictions = PlacesApi.placeAutocomplete(context, input)
            .location(location)
            .language("pt-BR")
            .radius(1000)
            .awaitIgnoreError();

        return assemblyPrediction(autocompletePredictions);
    }

    private List assemblyPrediction(AutocompletePrediction[] autoCompletePredictions) {
        List<Prediction> predictions = new ArrayList<>();

        for (AutocompletePrediction autocompletePrediction : autoCompletePredictions) {
            String placeId = autocompletePrediction.placeId;
            String title = autocompletePrediction.terms[0].value;
            String subTitle = Arrays.stream(autocompletePrediction.terms).skip(1).map(term -> term.value).collect(Collectors.joining(", "));

            predictions.add(new Prediction(placeId, title, subTitle));
        }

        return predictions;
    }
}

Sometimes we get this exception:

2017-04-19 19:11:03,290 [XNIO-3 task-914] DEBUG org.springframework.web.servlet.DispatcherServlet - Last-Modified value for [/predictions] is: -1
2017-04-19 19:11:03,290 [XNIO-3 task-914] INFO  com.application.services.GoogleMapsAPIService - GoogleMapsAPIService.placeAutoComplete
2017-04-19 19:11:03,290 [XNIO-3 task-914] DEBUG com.application.services.GoogleMapsAPIService -  > Passenger: [7459]
2017-04-19 19:11:03,291 [XNIO-3 task-914] DEBUG com.application.services.GoogleMapsAPIService -  > Input:     [Alameda santos 2081]
2017-04-19 19:11:03,291 [XNIO-3 task-914] DEBUG com.application.services.GoogleMapsAPIService -  > Latitude:  [-23.52488881961586]
2017-04-19 19:11:03,291 [XNIO-3 task-914] DEBUG com.application.services.GoogleMapsAPIService -  > Longitude: [-46.67578987777233]
2017-04-19 19:11:03,291 [XNIO-3 task-914] DEBUG com.application.services.GoogleMapsAPIService -  > Language:  [pt-BR]
2017-04-19 19:11:03,291 [XNIO-3 task-914] DEBUG com.application.services.GoogleMapsAPIService -  > Radius:    [1000]
2017-04-19 19:11:03,291 [XNIO-3 task-914] DEBUG org.springframework.web.servlet.DispatcherServlet - Could not complete request
org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.OutOfMemoryError: unable to create new native thread
        at org.springframework.web.servlet.DispatcherServlet.triggerAfterCompletionWithError(DispatcherServlet.java:1305)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:979)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:895)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:967)
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:858)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:843)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
        at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
        at org.springframework.boot.actuate.autoconfigure.EndpointWebMvcAutoConfiguration$ApplicationContextHeaderFilter.doFilterInternal(EndpointWebMvcAutoConfiguration.java:281)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
        at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

Caused by: java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:714)
        at com.google.maps.internal.RateLimitExecutorService.<init>(RateLimitExecutorService.java:63)
        at com.google.maps.OkHttpRequestHandler.<init>(OkHttpRequestHandler.java:49)
        at com.google.maps.GeoApiContext.<init>(GeoApiContext.java:88)
        at com.application.services.GoogleMapsAPIService.placeAutoComplete(GoogleMapsAPIService.java:175)
        at com.application.controllers.PredictionsController.predictions(PredictionsController.java:31)

Stacktrace: stacktrace.txt

Do you know what could causes that?

[domesticmouse]

The simplistic answer is that your server is running out of RAM. You probably need to run your server with some diagnostic logging to figure out if you are leaking resources somewhere. A temporary stopgap would be to increase the amount of memory available to the process.

[pooi-pooi]

@domesticmouse About the problem, that happens because my OS don't have more thread to 'instantiate'. My application reach out the operation system thread limit.

However, today I watched the JVM with VisualVM and I get a lot of RateLimitExecutorDelayThread that stills on "PARK". Thats happens on Undertow and Jetty.

Seeing #232 and #207 , to me it looks like it has some problem that involves the thread. Do you have any suggestion?

[domesticmouse]

I don't have any immediate fixes, but I would ask you to test the latest version as I think we may have made some changes between .17 and .20 that may impact this behavior. My understanding of #232 is that we are merely seeing that the OkHttp thread isn't being closed out on shutdown. And I'm unsure how #207 is akin to what you are currently seeing.

My suspicion is that you need to up the number of threads your system is configured to allow, or spread the workload over more machines.

[mkuehle]

This memory leak is still present in version .20. :/

The opened threads didn't close. They are still active after all my google maps requests are done. In the attached screenshot there are 100 requests. All threads are stille open. If the number of requests are increased the out of memory exception because of the active threads will be thrown.

[thpoiani]

I solve that implementing a Singleton to GeoApiContext instance

On May 31, 2017 12:20, "Markus Kühle" [email protected] wrote:

This memory leak is still present in version .20. :/

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/googlemaps/google-maps-services-java/issues/261#issuecomment-305221295, or mute the thread https://github.com/notifications/unsubscribe-auth/ABxiwRF-6Sjh1wJWtoWwHaGEaOq6ZiNuks5r_YVUgaJpZM4NCdZd .

[domesticmouse]

Re-opening to give myself a reminder that I need to fix this errant behaviour.

[domesticmouse]

This will be fixed in the next release version

[inctom]

Many thanks for fixing this. We have been fighting with that also. Can you guys tell when the next release is due? I am kind of waiting for this fix.

[domesticmouse]

Release is in process. Should turn up on maven in about a day.

[fpoolev]

This is still an issue as of version 0.2.4 using the Ok HTTP request handler. Use a singleton google maps API configuration singleton as a workaround.

[stafichuk]

I'm also facing this issue. The root cause is following. We are creating and starting a thread in a constructor of class RateLimitExecutorService. Here it is:

public RateLimitExecutorService() {
    setQueriesPerSecond(DEFAULT_QUERIES_PER_SECOND);
    Thread delayThread = new Thread(this);
    delayThread.setDaemon(true);
    delayThread.setName("RateLimitExecutorDelayThread");
    delayThread.start();
}

run method of this class looks like this:

@Override
public void run() {
  try {
    while (!delegate.isShutdown()) {
      this.rateLimiter.acquire();
      Runnable r = queue.take();
      delegate.execute(r);
    }
  } catch (InterruptedException ie) {
    LOG.info("Interrupted", ie);
  }
}

There are two problems with this method:

1. RateLimitExecutorDelayThread should end when the delegate is shut down but the thing is RateLimitExecutorService#shutdown method and thus shutdown() method of the delegate is never called.
2. If we call shutdown() method but the queue is empty we will wait forever in queue.take().

This issue manifests itself when you are restarting an application multiple times inside a single JVM without restarting JVM itself. For example, it can happen in application servers like Tomcat or in Play Framework's dev mode, which was my case.

[ptahchiev]

Just to let you know - i'm using 0.2.7 from maven repo and I still see the same warning:

WARNING: The web application [storefront] appears to have started a thread named [RateLimitExecutorDelayThread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
 [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
 [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2062)
 [email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
 app//com.google.maps.internal.RateLimitExecutorService.run(RateLimitExecutorService.java:78)
 [email protected]/java.lang.Thread.run(Thread.java:844)

[ptahchiev]

@domesticmouse I also see one more about Rate Limited Dispatcher:

WARNING: The web application [storefront] appears to have started a thread named [Rate Limited Dispatcher] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
 [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
 [email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
 [email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
 [email protected]/java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:920)
 [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1092)
 [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
 [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
 [email protected]/java.lang.Thread.run(Thread.java:844)

[ptahchiev]

Here's my whole stacktrace:

WARNING: The web application [storefront] appears to have started a thread named [RateLimitExecutorDelayThread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
 [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
 [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2062)
 [email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
 app//com.google.maps.internal.RateLimitExecutorService.run(RateLimitExecutorService.java:78)
 [email protected]/java.lang.Thread.run(Thread.java:844)
Jun 24, 2018 1:28:48 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [storefront] appears to have started a thread named [Rate Limited Dispatcher] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
 [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
 [email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
 [email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
 [email protected]/java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:920)
 [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1092)
 [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
 [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
 [email protected]/java.lang.Thread.run(Thread.java:844)
Jun 24, 2018 1:28:48 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [storefront] appears to have started a thread named [RateLimitExecutorDelayThread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
 [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
 [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2062)
 [email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
 app//com.google.maps.internal.RateLimitExecutorService.run(RateLimitExecutorService.java:78)
 [email protected]/java.lang.Thread.run(Thread.java:844)
Jun 24, 2018 1:28:48 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [storefront] appears to have started a thread named [Rate Limited Dispatcher] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
 [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
 [email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
 [email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
 [email protected]/java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:920)
 [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1092)
 [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
 [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
 [email protected]/java.lang.Thread.run(Thread.java:844)
Jun 24, 2018 1:28:48 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [storefront] appears to have started a thread named [RateLimitExecutorDelayThread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
 [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
 [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2062)
 [email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
 app//com.google.maps.internal.RateLimitExecutorService.run(RateLimitExecutorService.java:78)
 [email protected]/java.lang.Thread.run(Thread.java:844)
Jun 24, 2018 1:28:48 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [storefront] appears to have started a thread named [Rate Limited Dispatcher] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
 [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
 [email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
 [email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
 [email protected]/java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:920)
 [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1092)
 [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
 [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
 [email protected]/java.lang.Thread.run(Thread.java:844)
Jun 24, 2018 1:28:48 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [storefront] appears to have started a thread named [OkHttp maps.googleapis.com] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 [email protected]/java.net.SocketInputStream.socketRead0(Native Method)
 [email protected]/java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
 [email protected]/java.net.SocketInputStream.read(SocketInputStream.java:171)
 [email protected]/java.net.SocketInputStream.read(SocketInputStream.java:141)
 [email protected]/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:425)
 [email protected]/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:65)
 [email protected]/sun.security.ssl.SSLSocketImpl.bytesInCompletePacket(SSLSocketImpl.java:918)
 [email protected]/sun.security.ssl.AppInputStream.read(AppInputStream.java:144)
 app//okio.Okio$2.read(Okio.java:139)
 app//okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
 app//okio.RealBufferedSource.request(RealBufferedSource.java:67)
 app//okio.RealBufferedSource.require(RealBufferedSource.java:60)
 app//okhttp3.internal.http2.Http2Reader.nextFrame(Http2Reader.java:95)
 app//okhttp3.internal.http2.Http2Connection$ReaderRunnable.execute(Http2Connection.java:566)
 app//okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
 [email protected]/java.lang.Thread.run(Thread.java:844)
Jun 24, 2018 1:28:48 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [storefront] appears to have started a thread named [OkHttp ConnectionPool] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 [email protected]/java.lang.Object.wait(Native Method)
 [email protected]/java.lang.Object.wait(Object.java:474)
 app//okhttp3.ConnectionPool$1.run(ConnectionPool.java:67)
 [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
 [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
 [email protected]/java.lang.Thread.run(Thread.java:844)
Jun 24, 2018 1:28:48 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [storefront] appears to have started a thread named [RateLimitExecutorDelayThread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
 [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
 [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2062)
 [email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
 app//com.google.maps.internal.RateLimitExecutorService.run(RateLimitExecutorService.java:78)
 [email protected]/java.lang.Thread.run(Thread.java:844)
Jun 24, 2018 1:28:48 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [storefront] appears to have started a thread named [Rate Limited Dispatcher] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
 [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
 [email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
 [email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
 [email protected]/java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:920)
 [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1092)
 [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
 [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
 [email protected]/java.lang.Thread.run(Thread.java:844)
Jun 24, 2018 1:28:48 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [storefront] appears to have started a thread named [OkHttp maps.googleapis.com] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 [email protected]/java.net.SocketInputStream.socketRead0(Native Method)
 [email protected]/java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
 [email protected]/java.net.SocketInputStream.read(SocketInputStream.java:171)
 [email protected]/java.net.SocketInputStream.read(SocketInputStream.java:141)
 [email protected]/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:425)
 [email protected]/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:65)
 [email protected]/sun.security.ssl.SSLSocketImpl.bytesInCompletePacket(SSLSocketImpl.java:918)
 [email protected]/sun.security.ssl.AppInputStream.read(AppInputStream.java:144)
 app//okio.Okio$2.read(Okio.java:139)
 app//okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
 app//okio.RealBufferedSource.request(RealBufferedSource.java:67)
 app//okio.RealBufferedSource.require(RealBufferedSource.java:60)
 app//okhttp3.internal.http2.Http2Reader.nextFrame(Http2Reader.java:95)
 app//okhttp3.internal.http2.Http2Connection$ReaderRunnable.execute(Http2Connection.java:566)
 app//okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
 [email protected]/java.lang.Thread.run(Thread.java:844)
Jun 24, 2018 1:28:48 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [storefront] appears to have started a thread named [OkHttp ConnectionPool] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 [email protected]/java.lang.Object.wait(Native Method)
 [email protected]/java.lang.Object.wait(Object.java:474)
 app//okhttp3.ConnectionPool$1.run(ConnectionPool.java:67)
 [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
 [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
 [email protected]/java.lang.Thread.run(Thread.java:844)
Jun 24, 2018 1:28:48 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [storefront] appears to have started a thread named [RateLimitExecutorDelayThread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
 [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
 [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2062)
 [email protected]/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
 app//com.google.maps.internal.RateLimitExecutorService.run(RateLimitExecutorService.java:78)
 [email protected]/java.lang.Thread.run(Thread.java:844)
Jun 24, 2018 1:28:48 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [storefront] appears to have started a thread named [Rate Limited Dispatcher] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
 [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
 [email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
 [email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
 [email protected]/java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:920)
 [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1092)
 [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
 [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
 [email protected]/java.lang.Thread.run(Thread.java:844)
Jun 24, 2018 1:28:48 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [storefront] appears to have started a thread named [OkHttp maps.googleapis.com] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 [email protected]/java.net.SocketInputStream.socketRead0(Native Method)
 [email protected]/java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
 [email protected]/java.net.SocketInputStream.read(SocketInputStream.java:171)
 [email protected]/java.net.SocketInputStream.read(SocketInputStream.java:141)
 [email protected]/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:425)
 [email protected]/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:65)
 [email protected]/sun.security.ssl.SSLSocketImpl.bytesInCompletePacket(SSLSocketImpl.java:918)
 [email protected]/sun.security.ssl.AppInputStream.read(AppInputStream.java:144)
 app//okio.Okio$2.read(Okio.java:139)
 app//okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
 app//okio.RealBufferedSource.request(RealBufferedSource.java:67)
 app//okio.RealBufferedSource.require(RealBufferedSource.java:60)
 app//okhttp3.internal.http2.Http2Reader.nextFrame(Http2Reader.java:95)
 app//okhttp3.internal.http2.Http2Connection$ReaderRunnable.execute(Http2Connection.java:566)
 app//okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
 [email protected]/java.lang.Thread.run(Thread.java:844)
Jun 24, 2018 1:28:48 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [storefront] appears to have started a thread named [OkHttp ConnectionPool] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 [email protected]/java.lang.Object.wait(Native Method)
 [email protected]/java.lang.Object.wait(Object.java:474)
 app//okhttp3.ConnectionPool$1.run(ConnectionPool.java:67)
 [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
 [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
 [email protected]/java.lang.Thread.run(Thread.java:844)

[domesticmouse]

Version 0.2.8 is in the process of being uploaded through maven, tho I doubt it changes the status of the error you are seeing. I'll look into this once I get some time. Thanks for the report!

[apjanke]

Question: my impression from reading the code base was that GeoApiContext was intended to be used as a Singleton, or at least one instance per account/API key, otherwise the rate limiting wouldn’t work correctly, since Google’s rate limiting aggregates queries on a per-account or per-key basis. Are you expecting to have multiple GeoApiContext objects active simultaneously?

@pooi-pooi: I don’t see a shutdown() call in your example code. Am I missing something? I think you will have to call shutdown regardless of how the APi is defined, since finalizers can’t be relied on.

[domesticmouse]

Yup it is meant to be a singleton. There are advanced use cases where multiple contexts make sense, thus the lack of language level enforcement of the singleton pattern. On Mon, 2 Jul 2018 at 3:38 pm, Andrew Janke [email protected] wrote:

Question: my impression from reading the code base was that GeoApiContext was intended to be used as a Singleton, or at least one instance per account/API key, otherwise the rate limiting wouldn’t work correctly, since Google’s rate limiting aggregates queries on a per-account or per-key basis. Are you expecting to have multiple GeoApiContext objects active simultaneously?

@pooi-pooi https://github.com/pooi-pooi: I don’t see a shutdown() call in your example code. Am I missing something? I think you will have to call shutdown regardless of how the APi is defined, since finalizers can’t be relied on.

— You are receiving this because you modified the open/close state.

Reply to this email directly, view it on GitHub https://github.com/googlemaps/google-maps-services-java/issues/261#issuecomment-401677369, or mute the thread https://github.com/notifications/unsubscribe-auth/AAB3Jy7DOupLJdj1xgXM_N16RG5n7m0Iks5uCbHCgaJpZM4NCdZd .

[apjanke]

Here's another data point. I've added exercise-linger and exercise-mt-separate-contexts subcommands to my gmsj-cli tool to reproduce and debug this.

The exercise-linger case just creates a bunch of GeoApiContext objects in serial, running 100 queries on each, and shutting each down. It tops out at about 80 open threads. That seems more than necessary, but not catastrophic. The "Live threads" chart in the upper right is the one to look at.

./bin/gmsj-cli exercise-linger

The exercise-mt-separate-contexts case fires up 100 worker threads, and has them each spam queries through their own GeoApiContext as fast as the individual rate limiters will let them. This tops out around 1,000 live threads before eventually hitting the API's internal rate limit and shutting me down.

./bin/gmsj-cli exercise-mt-separate-contexts

(BTW, ha ha, isn't it cute how JConsole's window is a picture of a Mac window UI with its own title bar and window chrome?)

And here's the multi-threaded "dirty" result, where the worker threads do not call shutdown().

./bin/gmsj-cli exercise-mt-separate-contexts --dirty

That hit my rate limit pretty quick.

If you do a bunch of short-lived worker threads that only do a few queries each, and fail to call shutdown(), then it's off to the races. This would be more like the scenario of a server process that launches a new worker and a new GeoApiContext on every service call.

./bin/gmsj-cli exercise-mt-separate-contexts --dirty -n 10000 -p 10 -D 1000

With shutdown() calls, it steadies out at about 500 threads for me.

./bin/gmsj-cli exercise-mt-separate-contexts --dirty -n 10000 -p 10 -D 1000

While this didn't even bring my desktop to my knees, it does look like a thread leak.

Skimming through the source, I don't see anything that's actively managing a worker pool size. It looks like RateLimitExecutorService is just creating fresh threads for queries as they come in. Am I missing something?

[apjanke]

Skimming through the source, I don't see anything that's actively managing a worker pool size. It looks like RateLimitExecutorService is just creating fresh threads for queries as they come in.

I totally misread the source; it is in fact using a worker pool.

@ptahchiev I'm unable to reproduce the leak of threads from the main google-maps-services-java classes. Are you sure your app is calling GeoApiContext.shutdown() when it is done with them, or re-using a shared singleton GeoApiContext?

I am seeing thread growth up to about a steady state of 800 threads when I run a bunch of multi-threaded queries using their own GeoApiContext objects under the current code.

But looking at it in VisualVM, they all seem to be OkHttp worker pool threads. And they do seem to get cleaned up once there's some memory pressure happening. Plus, once I stop issuing new queries, they eventually all go away on their own.

This was after 1000 separate GeoApiContexts were built and used.

So I don't think there's a thread leak in google-maps-services-java per se any more, at least under simple usage scenarios.

Maybe a finalizer should be added to GeoApiContext to help support the scenario where the application is shut down from the outside by a web service container?

[domesticmouse]

I'm going to close this on the basis of @apjanke's thread eviction addition in #455. If this is insufficient, please comment to re-open and we will look what else we can do to deal with the issue.

#455 is included in Version 0.2.9, which is rolling out now.

[barrychapman]

This is still an issue, we get it in a few applications and causes Tomcat to crash

[domesticmouse]

@barrychapman which version of the library are you using?

[barrychapman]

Sorry for the delay.

                    <groupId>com.google.maps</groupId>
		<artifactId>google-maps-services</artifactId>
		<version>0.9.1</version>

[barrychapman]

Some of the errors in catalina.out

04-Jan-2019 06:30:17.889 WARNING [http-apr-8080-exec-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [shuttle] appears to have started a thread named [RateLimitExecutorDelayThread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
 java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
 com.google.maps.internal.RateLimitExecutorService.run(RateLimitExecutorService.java:78)
 java.lang.Thread.run(Thread.java:748)
04-Jan-2019 06:30:17.914 WARNING [http-apr-8080-exec-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [shuttle] appears to have started a thread named [Rate Limited Dispatcher] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
 java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
 java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
 java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)
 java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 java.lang.Thread.run(Thread.java:748)
04-Jan-2019 06:30:17.939 WARNING [http-apr-8080-exec-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [shuttle] appears to have started a thread named [RateLimitExecutorDelayThread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
 java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
 com.google.maps.internal.RateLimitExecutorService.run(RateLimitExecutorService.java:78)
 java.lang.Thread.run(Thread.java:748)
04-Jan-2019 06:30:17.966 WARNING [http-apr-8080-exec-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [shuttle] appears to have started a thread named [RateLimitExecutorDelayThread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
 java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
 com.google.maps.internal.RateLimitExecutorService.run(RateLimitExecutorService.java:78)
 java.lang.Thread.run(Thread.java:748)
04-Jan-2019 06:30:17.992 WARNING [http-apr-8080-exec-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [shuttle] appears to have started a thread named [Rate Limited Dispatcher] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
 java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
 java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
 java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)
 java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 java.lang.Thread.run(Thread.java:748)
04-Jan-2019 06:30:18.018 WARNING [http-apr-8080-exec-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [shuttle] appears to have started a thread named [RateLimitExecutorDelayThread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
 java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
 com.google.maps.internal.RateLimitExecutorService.run(RateLimitExecutorService.java:78)
 java.lang.Thread.run(Thread.java:748)
04-Jan-2019 06:30:18.042 WARNING [http-apr-8080-exec-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [shuttle] appears to have started a thread named [RateLimitExecutorDelayThread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
 java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
 com.google.maps.internal.RateLimitExecutorService.run(RateLimitExecutorService.java:78)
 java.lang.Thread.run(Thread.java:748)

The -only- place in my application that RateLimitExecutorService is invoked is inside the GoogleApiContext file.

[anshul35]

Getting the same issue :(

[anshul35]

Exception in thread "Rate Limited Dispatcher" java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:717) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957) at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1025) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Exception

[tguless]

Apparently if you are going to do this in your code like the docs specify:

       GeoApiContext context = new GeoApiContext.Builder().disableRetries()
                .apiKey(googleApiKey)
                .build();
        GeocodingResult[] results = GeocodingApi.geocode(context, address).await();

You must also do this: context.shutdown();

Which the docs do not specify.

This caused us to take major outages today as it would creep up under heavy workloads that did not surface during our regular testing.

Here are the misleading docs

Most of us like TLDR; not sure where the context.shutdown(); requirement is hidden in the docs.

Major pain in my ass. Thank you so much Google maps team.

@thpoiani's solution is what saved us in the end. Thanks for the recommendation.

For example, if you have a Spring Boot app, you should do this in one of you configuration classes:

@Bean
public GeoApiContext geoApiContext (@Value("${myGooleApiKey}") String googleApiKey) {

    GeoApiContext context = new GeoApiContext.Builder().disableRetries()
            .apiKey(googleApiKey)
            .build();
    return context;
}

And in the class where you want to issue a GeocodeAPI call, do this to get a reference to the singleton:

@Autowired
GeoApiContext geoApiContext;

What the code should instead provide is an out of the box singleton implementation which prevents this issue from happening (ties up a single thread eternally, instead of eventually causing the server to run out of threads).

What should also happen is that the docs should clearly state that the wheels of your server are going to come off if you do not also do a context.shutdown() if the above supplied code is going to be continuously called.

[barrywilks7]

Version 1.0.1 is still suffering a thread leak.

Even using the GeoApiContext as a singleton, a "Rate Limited Dispatcher" thread is created for each call and not cleared up (there is however only one RateLimitExecutorDelayThread)

I have had to resort to creating and destroying the GeoApiContext everytime we call the API, in a try-with-resources statement to get around this problem. This doesn't seem like a very efficient solution:

    try(GeoApiContext context = new GeoApiContext.Builder().apiKey(apiKey).build())
    {
        GeocodingApiRequest request = GeocodingApi.geocode(context,address)
                .language(language).region(region);
        GeocodingResult[] results = request.await();
        // process results
    }

a thread dump shows all the Rate Limited Dispatcher threads as follows:

"Rate Limited Dispatcher" Id=286 in WAITING on lock=java.util.concurrent.SynchronousQueue$TransferStack@7f71d2e6
    at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
    at [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
    at [email protected]/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
    at [email protected]/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
    at [email protected]/java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:920)
    at [email protected]/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
    at [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
    at [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at [email protected]/java.lang.Thread.run(Thread.java:829)

    Locked synchronizers: count = 0

[Kobee1203]

I noticed a problem when instantiating GeoApiContext.Builder.

I want to use my own RequestHandler.Builder. So I used the GeoApiContext.Builder.requestHandlerBuilder(RequestHandler.Builder) method:

try (GeoApiContext context = new GeoApiContext.Builder()
                   .apiKey(apiKey)
                    .requestHandlerBuilder(new MyRequestHandler.Builder())
                    .build()) {
    ...
}

This creates threads that are never closed. If we look at the details, we see that new GeoApiContext.Builder() creates a new OkHttpRequestHandler.Builder:

public Builder() {
    requestHandlerBuilder(new OkHttpRequestHandler.Builder());
}

new OkHttpRequestHandler.Builder() creates a new RateLimitExecutorService:

public Builder() {
      builder = new OkHttpClient.Builder();
      rateLimitExecutorService = new RateLimitExecutorService();
      ...
}

new RateLimitExecutorService() creates a new Thread:

public RateLimitExecutorService() {
    setQueriesPerSecond(DEFAULT_QUERIES_PER_SECOND);
    delayThread = new Thread(this);
    delayThread.setDaemon(true);
    delayThread.setName("RateLimitExecutorDelayThread");
    delayThread.start();
}

When we then call the GeoApiContext.Builder.requestHandlerBuilder(RequestHandler.Builder) method, it replaces the already present RequestHandler.Builder (instance of OkHttpRequestHandler.Builder) which has already created the Thread.

The GeoApiContext.Builder.build() method uses the last RequestHandler.Builder to create the RequestHandler.

When the GeoApiContext.shutdown() method is called, it calls the shutdown() method of the last RequestHandler:

public void shutdown() {
    requestHandler.shutdown();
}

The Threads created by new OkHttpRequestHandler.Builder() are therefore never closed.

After a while, I get the following exception: java.lang.OutOfMemoryError: Unable to create new native thread

A workaround is to not use the GeoApiContext.Builder.requestHandlerBuilder(RequestHandler.Builder) method and to use the GeoApiContext.Builder(RequestHandler.Builder) constructor:

try (final GeoApiContext context = new GeoApiContext.Builder(new MyRequestHandler.Builder())
                    .apiKey(apiKey)
                    .build()) {
    ...
}

I see 2 possible solutions to correct the problem:

• Delete the GeoApiContext.Builder.requestHandlerBuilder(RequestHandler.Builder) method, and use only the constructor with parameter
• Move the constructor code from OkHttpRequestHandler.Builder to the build() method