flow icon indicating copy to clipboard operation
flow copied to clipboard

Potential deadlock with Jetty

Open mperktold opened this issue 5 months ago • 7 comments

Description of the bug

We have several reports of our application not being able to shutdown. Apparently, some VaadinSessions stay alive and cannot be destroyed. Here the stack dumps of two such cases: StackTraces1.txt StackTraces2.txt

I found some common patterns in these dumps:

One of the threads blocks on a Jetty semaphore while reading the request content of an UIDL request. Note that this thread holds the lock on the VaadinSession while blocking.

[email protected]/jdk.internal.misc.Unsafe.park(Native Method)
[email protected]/java.util.concurrent.locks.LockSupport.park(Unknown Source)
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(Unknown Source)
[email protected]/java.util.concurrent.ForkJoinPool.unmanagedBlock(Unknown Source)
[email protected]/java.util.concurrent.ForkJoinPool.managedBlock(Unknown Source)
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
app//org.eclipse.jetty.ee10.servlet.AsyncContentProducer$LockedSemaphore.acquire(AsyncContentProducer.java:393)
app//org.eclipse.jetty.ee10.servlet.BlockingContentProducer.nextChunk(BlockingContentProducer.java:119)
app//org.eclipse.jetty.ee10.servlet.HttpInput.read(HttpInput.java:245)
app//org.eclipse.jetty.ee10.servlet.HttpInput.read(HttpInput.java:226)
[email protected]/sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
[email protected]/sun.nio.cs.StreamDecoder.implRead(Unknown Source)
[email protected]/sun.nio.cs.StreamDecoder.lockedRead(Unknown Source)
[email protected]/sun.nio.cs.StreamDecoder.read(Unknown Source)
[email protected]/java.io.InputStreamReader.read(Unknown Source)
[email protected]/java.io.BufferedReader.read1(Unknown Source)
[email protected]/java.io.BufferedReader.implRead(Unknown Source)
[email protected]/java.io.BufferedReader.read(Unknown Source)
[email protected]/java.io.Reader.read(Unknown Source)
app//com.vaadin.flow.server.communication.ServerRpcHandler.getMessage(ServerRpcHandler.java:503)
app//com.vaadin.flow.server.communication.ServerRpcHandler.handleRpc(ServerRpcHandler.java:253)
app//com.vaadin.flow.server.communication.UidlRequestHandler.synchronizedHandleRequest(UidlRequestHandler.java:114)
app//com.vaadin.flow.server.SynchronizedRequestHandler.handleRequest(SynchronizedRequestHandler.java:40)
app//com.vaadin.flow.server.VaadinService.handleRequest(VaadinService.java:1584)
app//com.vaadin.flow.server.VaadinServlet.service(VaadinServlet.java:398)
app//jakarta.servlet.http.HttpServlet.service(HttpServlet.java:614)
app//org.eclipse.jetty.ee10.servlet.ServletHolder.handle(ServletHolder.java:736)

A second thread blocks on the VaadinSession while trying to close the websocket:

[email protected]/jdk.internal.misc.Unsafe.park(Native Method)
[email protected]/java.util.concurrent.locks.LockSupport.park(Unknown Source)
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Unknown Source)
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Unknown Source)
[email protected]/java.util.concurrent.locks.ReentrantLock$Sync.lock(Unknown Source)
[email protected]/java.util.concurrent.locks.ReentrantLock.lock(Unknown Source)
app//com.vaadin.flow.server.VaadinService.lockSession(VaadinService.java:792)
app//com.vaadin.flow.server.VaadinService.findOrCreateVaadinSession(VaadinService.java:839)
app//com.vaadin.flow.server.VaadinService.findVaadinSession(VaadinService.java:684)
app//com.vaadin.flow.server.communication.PushHandler.handleConnectionLost(PushHandler.java:408)
app//com.vaadin.flow.server.communication.PushHandler.connectionLost(PushHandler.java:368)
app//com.vaadin.flow.server.communication.PushAtmosphereHandler.onStateChange(PushAtmosphereHandler.java:62)
app//org.atmosphere.cpr.AsynchronousProcessor.invokeAtmosphereHandler(AsynchronousProcessor.java:538)
app//org.atmosphere.cpr.AsynchronousProcessor.completeLifecycle(AsynchronousProcessor.java:480)
app//org.atmosphere.cpr.AsynchronousProcessor.endRequest(AsynchronousProcessor.java:584)
app//org.atmosphere.websocket.DefaultWebSocketProcessor.close(DefaultWebSocketProcessor.java:639)
app//org.atmosphere.container.JSR356Endpoint.onClose(JSR356Endpoint.java:318)
[email protected]/java.lang.invoke.LambdaForm$DMH/0x000000001e1a4000.invokeVirtual(LambdaForm$DMH)
[email protected]/java.lang.invoke.LambdaForm$MH/0x000000001f292000.invoke(LambdaForm$MH)
[email protected]/java.lang.invoke.LambdaForm$MH/0x000000001ee44800.invoke_MT(LambdaForm$MH)
app//org.eclipse.jetty.ee10.websocket.jakarta.common.JakartaWebSocketFrameHandler.notifyOnClose(JakartaWebSocketFrameHandler.java:295)
app//org.eclipse.jetty.ee10.websocket.jakarta.common.JakartaWebSocketFrameHandler.onClose(JakartaWebSocketFrameHandler.java:267)
app//org.eclipse.jetty.ee10.websocket.jakarta.common.JakartaWebSocketFrameHandler.onFrame(JakartaWebSocketFrameHandler.java:255)
app//org.eclipse.jetty.websocket.core.WebSocketCoreSession$IncomingAdaptor.onFrame(WebSocketCoreSession.java:680)
app//org.eclipse.jetty.websocket.core.AbstractExtension.nextIncomingFrame(AbstractExtension.java:145)
app//org.eclipse.jetty.websocket.core.internal.PerMessageDeflateExtension.nextIncomingFrame(PerMessageDeflateExtension.java:239)
app//org.eclipse.jetty.websocket.core.internal.PerMessageDeflateExtension$IncomingFlusher$$Lambda/0x000000001e90bd08.onFrame(Unknown Source)
app//org.eclipse.jetty.websocket.core.util.DemandingFlusher.emitFrame(DemandingFlusher.java:143)
app//org.eclipse.jetty.websocket.core.internal.PerMessageDeflateExtension$IncomingFlusher.handle(PerMessageDeflateExtension.java:382)
app//org.eclipse.jetty.websocket.core.util.DemandingFlusher.process(DemandingFlusher.java:167)
app//org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:262)
app//org.eclipse.jetty.util.IteratingCallback.succeeded(IteratingCallback.java:401)
app//org.eclipse.jetty.websocket.core.util.DemandingFlusher.onFrame(DemandingFlusher.java:105)
app//org.eclipse.jetty.websocket.core.internal.PerMessageDeflateExtension.onFrame(PerMessageDeflateExtension.java:96)
app//org.eclipse.jetty.websocket.core.ExtensionStack.onFrame(ExtensionStack.java:113)
app//org.eclipse.jetty.websocket.core.WebSocketCoreSession.onFrame(WebSocketCoreSession.java:463)
app//org.eclipse.jetty.websocket.core.WebSocketConnection.onFrame(WebSocketConnection.java:254)
app//org.eclipse.jetty.websocket.core.WebSocketConnection.fillAndParse(WebSocketConnection.java:447)
app//org.eclipse.jetty.websocket.core.WebSocketConnection.onFillable(WebSocketConnection.java:332)
app//org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)
app//org.eclipse.jetty.http2.HTTP2StreamEndPoint.process(HTTP2StreamEndPoint.java:497)
app//org.eclipse.jetty.http2.HTTP2StreamEndPoint.processDataAvailable(HTTP2StreamEndPoint.java:484)
app//org.eclipse.jetty.http2.server.internal.ServerHTTP2StreamEndPoint.onDataAvailable(ServerHTTP2StreamEndPoint.java:40)
app//org.eclipse.jetty.http2.server.internal.HTTP2ServerConnection.onDataAvailable(HTTP2ServerConnection.java:158)
app//org.eclipse.jetty.http2.server.HTTP2ServerConnectionFactory$HTTPServerSessionListener.onDataAvailable(HTTP2ServerConnectionFactory.java:153)
app//org.eclipse.jetty.http2.HTTP2Stream.notifyDataAvailable(HTTP2Stream.java:861)
app//org.eclipse.jetty.http2.HTTP2Stream.processData(HTTP2Stream.java:543)
app//org.eclipse.jetty.http2.HTTP2Stream.onData(HTTP2Stream.java:461)
app//org.eclipse.jetty.http2.HTTP2Stream.process(HTTP2Stream.java:368)
app//org.eclipse.jetty.http2.HTTP2Session.onData(HTTP2Session.java:280)
app//org.eclipse.jetty.http2.HTTP2Connection.onData(HTTP2Connection.java:246)
app//org.eclipse.jetty.http2.parser.BodyParser.notifyData(BodyParser.java:103)
app//org.eclipse.jetty.http2.parser.DataBodyParser.onData(DataBodyParser.java:145)
app//org.eclipse.jetty.http2.parser.DataBodyParser.onData(DataBodyParser.java:140)
app//org.eclipse.jetty.http2.parser.DataBodyParser.parse(DataBodyParser.java:106)
app//org.eclipse.jetty.http2.parser.Parser.parseBody(Parser.java:229)
app//org.eclipse.jetty.http2.parser.Parser.parse(Parser.java:156)
app//org.eclipse.jetty.http2.parser.ServerParser.parse(ServerParser.java:121)
app//org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:342)
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:512)
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:258)
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201)
app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311)
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:979)
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1209)
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1164)
[email protected]/java.lang.Thread.runWith(Unknown Source)
[email protected]/java.lang.Thread.run(Unknown Source)

A third thread also blocks on the VaadinSession while handling a connection loss, but this one comes from the HeartbeatInterception:

[email protected]/jdk.internal.misc.Unsafe.park(Native Method)
[email protected]/java.util.concurrent.locks.LockSupport.park(Unknown Source)
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Unknown Source)
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Unknown Source)
[email protected]/java.util.concurrent.locks.ReentrantLock$Sync.lock(Unknown Source)
[email protected]/java.util.concurrent.locks.ReentrantLock.lock(Unknown Source)
app//com.vaadin.flow.server.VaadinService.lockSession(VaadinService.java:798)
app//com.vaadin.flow.server.VaadinService.findOrCreateVaadinSession(VaadinService.java:845)
app//com.vaadin.flow.server.VaadinService.findVaadinSession(VaadinService.java:690)
app//com.vaadin.flow.server.communication.PushHandler.handleConnectionLost(PushHandler.java:414)
app//com.vaadin.flow.server.communication.PushHandler.connectionLost(PushHandler.java:368)
app//com.vaadin.flow.server.communication.PushAtmosphereHandler$AtmosphereResourceListener.onDisconnect(PushAtmosphereHandler.java:113)
app//org.atmosphere.cpr.AtmosphereResourceImpl.onDisconnect(AtmosphereResourceImpl.java:752)
app//org.atmosphere.cpr.AtmosphereResourceImpl.notifyListeners(AtmosphereResourceImpl.java:644)
app//org.atmosphere.cpr.AtmosphereResponseImpl.handleException(AtmosphereResponseImpl.java:732)
app//org.atmosphere.cpr.AtmosphereResponseImpl.access$1500(AtmosphereResponseImpl.java:57)
app//org.atmosphere.cpr.AtmosphereResponseImpl$Stream.write(AtmosphereResponseImpl.java:958)
app//org.atmosphere.cpr.AtmosphereResponseImpl.write(AtmosphereResponseImpl.java:805)
app//org.atmosphere.interceptor.HeartbeatInterceptor.lambda$clock$0(HeartbeatInterceptor.java:367)
app//org.atmosphere.interceptor.HeartbeatInterceptor$$Lambda/0x0000000021c1cf78.call(Unknown Source)
[email protected]/java.util.concurrent.FutureTask.run(Unknown Source)
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[email protected]/java.lang.Thread.runWith(Unknown Source)
[email protected]/java.lang.Thread.run(Unknown Source)

I'm not sure where things go wrong, but this does look a bit suspicious to me.

Expected behavior

The application should shut down without getting stuck in a deadlock.

Minimal reproducible example

Unfortunately, I don't have a reproducer. I hope that the stack traces are good enough.

Versions

  • Vaadin / Flow version: 24.4.10
  • Java version: Eclipse Temurin 21.0.3
  • OS version: Windows 11
  • Application Server (if applicable): Jetty 12.0.13

mperktold avatar Sep 11 '24 14:09 mperktold