An active request indefinitely delays the shutdown of an app that is using Reactor Netty

[wilkinsona]

Unlike Jetty, Tomcat, and Undertow, shutdown of an application that uses Reactor Netty is delayed by an active request. It appears to be delayed until the request completes. Looking at a thread dump during the delay, it appears to be caused by the destruction of ReactorResourceFactory:

"SpringContextShutdownHook" #16 prio=5 os_prio=31 tid=0x00007fda119d7800 nid=0x5803 waiting on condition [0x0000700008599000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000007175081c0> (a java.util.concurrent.CountDownLatch$Sync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
	at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:87)
	at reactor.core.publisher.Mono.block(Mono.java:1678)
	at org.springframework.http.client.reactive.ReactorResourceFactory.destroy(ReactorResourceFactory.java:225)
	at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:258)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:579)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:551)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:1091)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:512)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:1084)
	at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1060)
	at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1029)
	at org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:948)
	- locked <0x00000005c00d33a0> (a java.lang.Object)

[wilkinsona]

The fact that the delay appears to be indefinite (I have seen request handling with a 60 second sleep be allowed to complete) may be a Reactor Netty bug. The default quiet period is 2 seconds and the default timeout is 15 seconds. I think that should result in a 17 second delay before the connection's dropped.

[jhoglin]

I mentioned this issue to @violetagg on reactor-netty's Gitter, and was informed about this test-case, this is to validate that the Netty indeed shuts down as expected. Maybe this can help someone to resolve the issue.

https://github.com/reactor/reactor-netty/blob/ca10139086abdeaf9fd9db7076b5511532e8326e/reactor-netty-http/src/test/java/reactor/netty/http/server/HttpServerTests.java#L1725

[philwebb]

The ReactorResourceFactory code has moved on a bit, it looks like this version matches the stacktrace.