flow
flow copied to clipboard
Potential deadlock with Jetty
Description of the bug
We have several reports of our application not being able to shutdown. Apparently, some VaadinSessions stay alive and cannot be destroyed. Here the stack dumps of two such cases: StackTraces1.txt StackTraces2.txt
I found some common patterns in these dumps:
One of the threads blocks on a Jetty semaphore while reading the request content of an UIDL request. Note that this thread holds the lock on the VaadinSession while blocking.
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)
[email protected]/java.util.concurrent.locks.LockSupport.park(Unknown Source)
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(Unknown Source)
[email protected]/java.util.concurrent.ForkJoinPool.unmanagedBlock(Unknown Source)
[email protected]/java.util.concurrent.ForkJoinPool.managedBlock(Unknown Source)
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
app//org.eclipse.jetty.ee10.servlet.AsyncContentProducer$LockedSemaphore.acquire(AsyncContentProducer.java:393)
app//org.eclipse.jetty.ee10.servlet.BlockingContentProducer.nextChunk(BlockingContentProducer.java:119)
app//org.eclipse.jetty.ee10.servlet.HttpInput.read(HttpInput.java:245)
app//org.eclipse.jetty.ee10.servlet.HttpInput.read(HttpInput.java:226)
[email protected]/sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
[email protected]/sun.nio.cs.StreamDecoder.implRead(Unknown Source)
[email protected]/sun.nio.cs.StreamDecoder.lockedRead(Unknown Source)
[email protected]/sun.nio.cs.StreamDecoder.read(Unknown Source)
[email protected]/java.io.InputStreamReader.read(Unknown Source)
[email protected]/java.io.BufferedReader.read1(Unknown Source)
[email protected]/java.io.BufferedReader.implRead(Unknown Source)
[email protected]/java.io.BufferedReader.read(Unknown Source)
[email protected]/java.io.Reader.read(Unknown Source)
app//com.vaadin.flow.server.communication.ServerRpcHandler.getMessage(ServerRpcHandler.java:503)
app//com.vaadin.flow.server.communication.ServerRpcHandler.handleRpc(ServerRpcHandler.java:253)
app//com.vaadin.flow.server.communication.UidlRequestHandler.synchronizedHandleRequest(UidlRequestHandler.java:114)
app//com.vaadin.flow.server.SynchronizedRequestHandler.handleRequest(SynchronizedRequestHandler.java:40)
app//com.vaadin.flow.server.VaadinService.handleRequest(VaadinService.java:1584)
app//com.vaadin.flow.server.VaadinServlet.service(VaadinServlet.java:398)
app//jakarta.servlet.http.HttpServlet.service(HttpServlet.java:614)
app//org.eclipse.jetty.ee10.servlet.ServletHolder.handle(ServletHolder.java:736)
A second thread blocks on the VaadinSession while trying to close the websocket:
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)
[email protected]/java.util.concurrent.locks.LockSupport.park(Unknown Source)
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Unknown Source)
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Unknown Source)
[email protected]/java.util.concurrent.locks.ReentrantLock$Sync.lock(Unknown Source)
[email protected]/java.util.concurrent.locks.ReentrantLock.lock(Unknown Source)
app//com.vaadin.flow.server.VaadinService.lockSession(VaadinService.java:792)
app//com.vaadin.flow.server.VaadinService.findOrCreateVaadinSession(VaadinService.java:839)
app//com.vaadin.flow.server.VaadinService.findVaadinSession(VaadinService.java:684)
app//com.vaadin.flow.server.communication.PushHandler.handleConnectionLost(PushHandler.java:408)
app//com.vaadin.flow.server.communication.PushHandler.connectionLost(PushHandler.java:368)
app//com.vaadin.flow.server.communication.PushAtmosphereHandler.onStateChange(PushAtmosphereHandler.java:62)
app//org.atmosphere.cpr.AsynchronousProcessor.invokeAtmosphereHandler(AsynchronousProcessor.java:538)
app//org.atmosphere.cpr.AsynchronousProcessor.completeLifecycle(AsynchronousProcessor.java:480)
app//org.atmosphere.cpr.AsynchronousProcessor.endRequest(AsynchronousProcessor.java:584)
app//org.atmosphere.websocket.DefaultWebSocketProcessor.close(DefaultWebSocketProcessor.java:639)
app//org.atmosphere.container.JSR356Endpoint.onClose(JSR356Endpoint.java:318)
[email protected]/java.lang.invoke.LambdaForm$DMH/0x000000001e1a4000.invokeVirtual(LambdaForm$DMH)
[email protected]/java.lang.invoke.LambdaForm$MH/0x000000001f292000.invoke(LambdaForm$MH)
[email protected]/java.lang.invoke.LambdaForm$MH/0x000000001ee44800.invoke_MT(LambdaForm$MH)
app//org.eclipse.jetty.ee10.websocket.jakarta.common.JakartaWebSocketFrameHandler.notifyOnClose(JakartaWebSocketFrameHandler.java:295)
app//org.eclipse.jetty.ee10.websocket.jakarta.common.JakartaWebSocketFrameHandler.onClose(JakartaWebSocketFrameHandler.java:267)
app//org.eclipse.jetty.ee10.websocket.jakarta.common.JakartaWebSocketFrameHandler.onFrame(JakartaWebSocketFrameHandler.java:255)
app//org.eclipse.jetty.websocket.core.WebSocketCoreSession$IncomingAdaptor.onFrame(WebSocketCoreSession.java:680)
app//org.eclipse.jetty.websocket.core.AbstractExtension.nextIncomingFrame(AbstractExtension.java:145)
app//org.eclipse.jetty.websocket.core.internal.PerMessageDeflateExtension.nextIncomingFrame(PerMessageDeflateExtension.java:239)
app//org.eclipse.jetty.websocket.core.internal.PerMessageDeflateExtension$IncomingFlusher$$Lambda/0x000000001e90bd08.onFrame(Unknown Source)
app//org.eclipse.jetty.websocket.core.util.DemandingFlusher.emitFrame(DemandingFlusher.java:143)
app//org.eclipse.jetty.websocket.core.internal.PerMessageDeflateExtension$IncomingFlusher.handle(PerMessageDeflateExtension.java:382)
app//org.eclipse.jetty.websocket.core.util.DemandingFlusher.process(DemandingFlusher.java:167)
app//org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:262)
app//org.eclipse.jetty.util.IteratingCallback.succeeded(IteratingCallback.java:401)
app//org.eclipse.jetty.websocket.core.util.DemandingFlusher.onFrame(DemandingFlusher.java:105)
app//org.eclipse.jetty.websocket.core.internal.PerMessageDeflateExtension.onFrame(PerMessageDeflateExtension.java:96)
app//org.eclipse.jetty.websocket.core.ExtensionStack.onFrame(ExtensionStack.java:113)
app//org.eclipse.jetty.websocket.core.WebSocketCoreSession.onFrame(WebSocketCoreSession.java:463)
app//org.eclipse.jetty.websocket.core.WebSocketConnection.onFrame(WebSocketConnection.java:254)
app//org.eclipse.jetty.websocket.core.WebSocketConnection.fillAndParse(WebSocketConnection.java:447)
app//org.eclipse.jetty.websocket.core.WebSocketConnection.onFillable(WebSocketConnection.java:332)
app//org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)
app//org.eclipse.jetty.http2.HTTP2StreamEndPoint.process(HTTP2StreamEndPoint.java:497)
app//org.eclipse.jetty.http2.HTTP2StreamEndPoint.processDataAvailable(HTTP2StreamEndPoint.java:484)
app//org.eclipse.jetty.http2.server.internal.ServerHTTP2StreamEndPoint.onDataAvailable(ServerHTTP2StreamEndPoint.java:40)
app//org.eclipse.jetty.http2.server.internal.HTTP2ServerConnection.onDataAvailable(HTTP2ServerConnection.java:158)
app//org.eclipse.jetty.http2.server.HTTP2ServerConnectionFactory$HTTPServerSessionListener.onDataAvailable(HTTP2ServerConnectionFactory.java:153)
app//org.eclipse.jetty.http2.HTTP2Stream.notifyDataAvailable(HTTP2Stream.java:861)
app//org.eclipse.jetty.http2.HTTP2Stream.processData(HTTP2Stream.java:543)
app//org.eclipse.jetty.http2.HTTP2Stream.onData(HTTP2Stream.java:461)
app//org.eclipse.jetty.http2.HTTP2Stream.process(HTTP2Stream.java:368)
app//org.eclipse.jetty.http2.HTTP2Session.onData(HTTP2Session.java:280)
app//org.eclipse.jetty.http2.HTTP2Connection.onData(HTTP2Connection.java:246)
app//org.eclipse.jetty.http2.parser.BodyParser.notifyData(BodyParser.java:103)
app//org.eclipse.jetty.http2.parser.DataBodyParser.onData(DataBodyParser.java:145)
app//org.eclipse.jetty.http2.parser.DataBodyParser.onData(DataBodyParser.java:140)
app//org.eclipse.jetty.http2.parser.DataBodyParser.parse(DataBodyParser.java:106)
app//org.eclipse.jetty.http2.parser.Parser.parseBody(Parser.java:229)
app//org.eclipse.jetty.http2.parser.Parser.parse(Parser.java:156)
app//org.eclipse.jetty.http2.parser.ServerParser.parse(ServerParser.java:121)
app//org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:342)
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:512)
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:258)
app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201)
app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311)
app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:979)
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1209)
app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1164)
[email protected]/java.lang.Thread.runWith(Unknown Source)
[email protected]/java.lang.Thread.run(Unknown Source)
A third thread also blocks on the VaadinSession while handling a connection loss, but this one comes from the HeartbeatInterception:
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)
[email protected]/java.util.concurrent.locks.LockSupport.park(Unknown Source)
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Unknown Source)
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Unknown Source)
[email protected]/java.util.concurrent.locks.ReentrantLock$Sync.lock(Unknown Source)
[email protected]/java.util.concurrent.locks.ReentrantLock.lock(Unknown Source)
app//com.vaadin.flow.server.VaadinService.lockSession(VaadinService.java:798)
app//com.vaadin.flow.server.VaadinService.findOrCreateVaadinSession(VaadinService.java:845)
app//com.vaadin.flow.server.VaadinService.findVaadinSession(VaadinService.java:690)
app//com.vaadin.flow.server.communication.PushHandler.handleConnectionLost(PushHandler.java:414)
app//com.vaadin.flow.server.communication.PushHandler.connectionLost(PushHandler.java:368)
app//com.vaadin.flow.server.communication.PushAtmosphereHandler$AtmosphereResourceListener.onDisconnect(PushAtmosphereHandler.java:113)
app//org.atmosphere.cpr.AtmosphereResourceImpl.onDisconnect(AtmosphereResourceImpl.java:752)
app//org.atmosphere.cpr.AtmosphereResourceImpl.notifyListeners(AtmosphereResourceImpl.java:644)
app//org.atmosphere.cpr.AtmosphereResponseImpl.handleException(AtmosphereResponseImpl.java:732)
app//org.atmosphere.cpr.AtmosphereResponseImpl.access$1500(AtmosphereResponseImpl.java:57)
app//org.atmosphere.cpr.AtmosphereResponseImpl$Stream.write(AtmosphereResponseImpl.java:958)
app//org.atmosphere.cpr.AtmosphereResponseImpl.write(AtmosphereResponseImpl.java:805)
app//org.atmosphere.interceptor.HeartbeatInterceptor.lambda$clock$0(HeartbeatInterceptor.java:367)
app//org.atmosphere.interceptor.HeartbeatInterceptor$$Lambda/0x0000000021c1cf78.call(Unknown Source)
[email protected]/java.util.concurrent.FutureTask.run(Unknown Source)
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[email protected]/java.lang.Thread.runWith(Unknown Source)
[email protected]/java.lang.Thread.run(Unknown Source)
I'm not sure where things go wrong, but this does look a bit suspicious to me.
Expected behavior
The application should shut down without getting stuck in a deadlock.
Minimal reproducible example
Unfortunately, I don't have a reproducer. I hope that the stack traces are good enough.
Versions
- Vaadin / Flow version: 24.4.10
- Java version: Eclipse Temurin 21.0.3
- OS version: Windows 11
- Application Server (if applicable): Jetty 12.0.13