Suggester rebuilds are blocking config updates and suggestion lookups

[aerofeev2k]

/projects/PROJECT/indexed or /suggest/rebuild REST triggers a rebuild of Suggester DB for the specified project. The rebuild happens from a single thread like below:

"ForkJoinPool-4-worker-19" #1570 daemon prio=5 os_prio=0 cpu=150944.65ms elapsed=155.79s tid=0x00007f206406f000 nid=0x2560b runnable  [0x00007f2557dc7000]
   java.lang.Thread.State: RUNNABLE
        at org.apache.lucene.util.fst.Builder$UnCompiledNode.prependOutput(Builder.java:683)
        at org.apache.lucene.util.fst.Builder.add(Builder.java:486)
        at org.apache.lucene.search.suggest.fst.WFSTCompletionLookup.build(WFSTCompletionLookup.java:130)
        at org.opengrok.suggest.SuggesterProjectData.build(SuggesterProjectData.java:266)
        at org.opengrok.suggest.SuggesterProjectData.build(SuggesterProjectData.java:253)
        at org.opengrok.suggest.SuggesterProjectData.rebuild(SuggesterProjectData.java:238)
        at org.opengrok.suggest.Suggester.lambda$getRebuildRunnable$2(Suggester.java:352)
        at org.opengrok.suggest.Suggester$$Lambda$1908/0x0000000800636440.run(Unknown Source)
        at java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec([email protected]/ForkJoinTask.java:1407)
        at java.util.concurrent.ForkJoinTask.doExec([email protected]/ForkJoinTask.java:290)
        at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec([email protected]/ForkJoinPool.java:1020)
        at java.util.concurrent.ForkJoinPool.scan([email protected]/ForkJoinPool.java:1656)
        at java.util.concurrent.ForkJoinPool.runWorker([email protected]/ForkJoinPool.java:1594)
        at java.util.concurrent.ForkJoinWorkerThread.run([email protected]/ForkJoinWorkerThread.java:183)

If while this is going on (and since it's just a single thread it's possible that a number of Suggester rebuilds have been queued this way, so this can go on for a while), someone tries to push a new configuration via REST, the push hangs waiting for HTTP 200, and the webapp is meanwhile hanging trying to schedule a refresh of Suggesters with the stack like below:

"configuration-3-thread-1" #877 prio=5 os_prio=0 cpu=368.51ms elapsed=1907.70s tid=0x00007f25e804e000 nid=0x1ca2b waiting on condition  [0x00007f266a094000]
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
        - parking to wait for  <0x000000041f677228> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:194)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt([email protected]/AbstractQueuedSynchronizer.java:885)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued([email protected]/AbstractQueuedSynchronizer.java:917)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/AbstractQueuedSynchronizer.java:1240)
        at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock([email protected]/ReentrantReadWriteLock.java:959)
        at org.opengrok.web.api.v1.suggester.provider.service.impl.SuggesterServiceImpl.refresh(SuggesterServiceImpl.java:161)
        at org.opengrok.web.api.v1.controller.ConfigurationController.lambda$set$0(ConfigurationController.java:82)
        at org.opengrok.web.api.v1.controller.ConfigurationController$$Lambda$1896/0x0000000800645840.call(Unknown Source)
        at org.opengrok.web.api.ApiTask.lambda$getCallable$0(ApiTask.java:197)
        at org.opengrok.web.api.ApiTask$$Lambda$1897/0x0000000800645c40.call(Unknown Source)
        at java.util.concurrent.FutureTask.run([email protected]/FutureTask.java:264)
        at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1128)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:628)
        at java.lang.Thread.run([email protected]/Thread.java:834)

At this time suggestion lookups in the webpage stop working completely hanging with this stack in the webapp:

"ajp-nio-0:0:0:0:0:0:0:1-8009-exec-7" #594 daemon prio=5 os_prio=0 cpu=1345.45ms elapsed=1932.68s tid=0x00007f27298f4000 nid=0x1b795 waiting on condition  [0x00007f25d35d5000]
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
        - parking to wait for  <0x000000041f677228> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:194)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt([email protected]/AbstractQueuedSynchronizer.java:885)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared([email protected]/AbstractQueuedSynchronizer.java:1009)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared([email protected]/AbstractQueuedSynchronizer.java:1324)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock([email protected]/ReentrantReadWriteLock.java:738)
        at org.opengrok.web.api.v1.suggester.provider.service.impl.SuggesterServiceImpl.getSuggestions(SuggesterServiceImpl.java:108)
        at org.opengrok.web.api.v1.controller.SuggesterController.getSuggestions(SuggesterController.java:123)
        at jdk.internal.reflect.GeneratedMethodAccessor146.invoke(Unknown Source)
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke([email protected]/DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke([email protected]/Method.java:566)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$$Lambda$1652/0x00000008006dc840.invoke(Unknown Source)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:475)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:397)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)

[vladak]

Possibly similar to #3468 except the suggestion lookups which should be probably covered by separate bug.