vertx-kafka-client icon indicating copy to clipboard operation
vertx-kafka-client copied to clipboard

Published 20 hours ago • verified publisher icon vert-x3

Blocked thread in KafkaConsumer.poll

Open sishbi opened this issue 11 months ago • 4 comments

Version

4.5.4

Context

Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 2748 ms, time limit is 2000 ms

io.vertx.core.VertxException: Thread blocked
	at java.base/jdk.internal.misc.Unsafe.unpark(Native Method)
	at java.base/java.util.concurrent.locks.LockSupport.unpark(LockSupport.java:181)
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.signalNext(AbstractQueuedSynchronizer.java:645)
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1060)
	at java.base/java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:494)
	at java.base/java.util.concurrent.LinkedBlockingQueue.signalNotEmpty(LinkedBlockingQueue.java:179)
	at java.base/java.util.concurrent.LinkedBlockingQueue.offer(LinkedBlockingQueue.java:423)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:123)
	at java.base/java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:785)
	at io.vertx.kafka.client.consumer.impl.KafkaReadStreamImpl.poll(KafkaReadStreamImpl.java:876)
	at io.vertx.kafka.client.consumer.impl.KafkaConsumerImpl.poll(KafkaConsumerImpl.java:683)
	at io.vertx.kafka.client.consumer.impl.KafkaConsumerImpl.poll(KafkaConsumerImpl.java:695)
	at com.kodypay.lychee.kafka.KafkaConsumer$3.invokeSuspend(KafkaConsumer.kt:64)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
	at kotlinx.coroutines.EventLoop.processUnconfinedEvent(EventLoop.common.kt:65)
	at kotlinx.coroutines.internal.DispatchedContinuationKt.resumeCancellableWith(DispatchedContinuation.kt:371)
	at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable(Cancellable.kt:26)
	at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable$default(Cancellable.kt:21)
	at kotlinx.coroutines.CoroutineStart.invoke(CoroutineStart.kt:88)
	at kotlinx.coroutines.AbstractCoroutine.start(AbstractCoroutine.kt:123)
	at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch(Builders.common.kt:52)
	at kotlinx.coroutines.BuildersKt.launch(Unknown Source)
	at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch$default(Builders.common.kt:43)
	at kotlinx.coroutines.BuildersKt.launch$default(Unknown Source)
	at io.vertx.kotlin.coroutines.CoroutineVerticle.start(CoroutineVerticle.kt:51)
	at io.vertx.core.impl.DeploymentManager.lambda$doDeploy$5(DeploymentManager.java:210)
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:279)
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:261)
	at io.vertx.core.impl.ContextInternal.lambda$runOnContext$0(ContextInternal.java:59)
	at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
	at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:1583)

Do you have a reproducer?

No

Steps to reproduce

  1. Start vertx application
  2. Deploy kafka consumer
  3. Wait

Extra

OS: Amazon Linux (Docker) running in EKS (k8s) JDK: 21, eclipse-temurin:21-jdk

sishbi avatar Mar 26 '24 08:03 sishbi

I see you reported another blocked error (DNS) based on Kotlin, I am wondering if Kotlin coroutines is not related to that instead of Vert.x

vietj avatar Mar 26 '24 09:03 vietj

I see you reported another blocked error (DNS) based on Kotlin, I am wondering if Kotlin coroutines is not related to that instead of Vert.x

For the SQL client issue https://github.com/eclipse-vertx/vertx-sql-client/issues/1429, the blocked thread is during a class-load triggered by a DNS lookup in Netty code

For this issue the blocked thread originated in io.vertx.kafka.client.consumer.impl.KafkaReadStreamImpl.poll invoking something on an executor

I don't know if Kotlin coroutines is making this worse or not - but as this is within the Vertx/Netty code there isn't much we can do from our code...

sishbi avatar Mar 26 '24 12:03 sishbi

Some more stacks: This is from 2 pods starting at the same time, so some of the stacks may be repeated...

pod1: Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 2388 ms, time limit is 2000 ms Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 3387 ms, time limit is 2000 ms pod2: Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 2817 ms, time limit is 2000 ms

io.vertx.core.VertxException: Thread blocked
	at java.base/java.lang.Thread.start0(Native Method)
	at java.base/java.lang.Thread.start(Thread.java:1553)
	at java.base/java.lang.System$2.start(System.java:2577)
	at java.base/jdk.internal.vm.SharedThreadContainer.start(SharedThreadContainer.java:152)
	at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:953)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1364)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:123)
	at java.base/java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:785)
	at io.vertx.kafka.client.consumer.impl.KafkaReadStreamImpl.submitTaskWhenStarted(KafkaReadStreamImpl.java:123)
	at io.vertx.kafka.client.consumer.impl.KafkaReadStreamImpl.start(KafkaReadStreamImpl.java:116)
	at io.vertx.kafka.client.consumer.impl.KafkaReadStreamImpl.subscribe(KafkaReadStreamImpl.java:443)
	at io.vertx.kafka.client.consumer.impl.KafkaConsumerImpl.subscribe(KafkaConsumerImpl.java:197)
	at io.vertx.kafka.client.consumer.impl.KafkaConsumerImpl.subscribe(KafkaConsumerImpl.java:186)
	at io.vertx.kafka.client.consumer.impl.KafkaConsumerImpl.subscribe(KafkaConsumerImpl.java:180)
	at com.kodypay.lychee.kafka.KafkaConsumer.<init>(KafkaConsumer.kt:57)
	at com.kodypay.cards.kafka.payments.SettlementConsumer.<init>(SettlementConsumer.kt:34)
	at com.kodypay.cards.kafka.KafkaVerticle.start(KafkaVerticle.kt:37)
	at io.vertx.kotlin.coroutines.CoroutineVerticle$start$1.invokeSuspend(CoroutineVerticle.kt:53)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.internal.DispatchedContinuationKt.resumeCancellableWith(DispatchedContinuation.kt:363)
	at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable(Cancellable.kt:26)
	at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable$default(Cancellable.kt:21)
	at kotlinx.coroutines.CoroutineStart.invoke(CoroutineStart.kt:88)
	at kotlinx.coroutines.AbstractCoroutine.start(AbstractCoroutine.kt:123)
	at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch(Builders.common.kt:52)
	at kotlinx.coroutines.BuildersKt.launch(Unknown Source)
	at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch$default(Builders.common.kt:43)
	at kotlinx.coroutines.BuildersKt.launch$default(Unknown Source)
	at io.vertx.kotlin.coroutines.CoroutineVerticle.start(CoroutineVerticle.kt:51)
	at io.vertx.core.impl.DeploymentManager.lambda$doDeploy$5(DeploymentManager.java:210)
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:279)
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:261)
	at io.vertx.core.impl.ContextInternal.lambda$runOnContext$0(ContextInternal.java:59)
	at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
	at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:1583)

io.vertx.core.VertxException: Thread blocked
	at java.base/jdk.internal.misc.Unsafe.unpark(Native Method)
	at java.base/java.util.concurrent.locks.LockSupport.unpark(LockSupport.java:181)
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.signalNext(AbstractQueuedSynchronizer.java:645)
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1060)
	at java.base/java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:494)
	at java.base/java.util.concurrent.LinkedBlockingQueue.signalNotEmpty(LinkedBlockingQueue.java:179)
	at java.base/java.util.concurrent.LinkedBlockingQueue.offer(LinkedBlockingQueue.java:423)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:123)
	at java.base/java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:785)
	at io.vertx.kafka.client.consumer.impl.KafkaReadStreamImpl.poll(KafkaReadStreamImpl.java:876)
	at io.vertx.kafka.client.consumer.impl.KafkaConsumerImpl.poll(KafkaConsumerImpl.java:683)
	at io.vertx.kafka.client.consumer.impl.KafkaConsumerImpl.poll(KafkaConsumerImpl.java:695)
	at com.kodypay.lychee.kafka.KafkaConsumer$3.invokeSuspend(KafkaConsumer.kt:65)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
	at kotlinx.coroutines.EventLoop.processUnconfinedEvent(EventLoop.common.kt:65)
	at kotlinx.coroutines.internal.DispatchedContinuationKt.resumeCancellableWith(DispatchedContinuation.kt:371)
	at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable(Cancellable.kt:26)
	at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable$default(Cancellable.kt:21)
	at kotlinx.coroutines.CoroutineStart.invoke(CoroutineStart.kt:88)
	at kotlinx.coroutines.AbstractCoroutine.start(AbstractCoroutine.kt:123)
	at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch(Builders.common.kt:52)
	at kotlinx.coroutines.BuildersKt.launch(Unknown Source)
	at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch$default(Builders.common.kt:43)
	at kotlinx.coroutines.BuildersKt.launch$default(Unknown Source)
	at io.vertx.kotlin.coroutines.CoroutineVerticle.start(CoroutineVerticle.kt:51)
	at io.vertx.core.impl.DeploymentManager.lambda$doDeploy$5(DeploymentManager.java:210)
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:279)
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:261)
	at io.vertx.core.impl.ContextInternal.lambda$runOnContext$0(ContextInternal.java:59)
	at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
	at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:1583)

io.vertx.core.VertxException: Thread blocked
	at java.base/jdk.internal.misc.Unsafe.unpark(Native Method)
	at java.base/java.util.concurrent.locks.LockSupport.unpark(LockSupport.java:181)
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.signalNext(AbstractQueuedSynchronizer.java:645)
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1060)
	at java.base/java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:494)
	at java.base/java.util.concurrent.LinkedBlockingQueue.signalNotEmpty(LinkedBlockingQueue.java:179)
	at java.base/java.util.concurrent.LinkedBlockingQueue.offer(LinkedBlockingQueue.java:423)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:123)
	at java.base/java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:785)
	at io.vertx.kafka.client.consumer.impl.KafkaReadStreamImpl.poll(KafkaReadStreamImpl.java:876)
	at io.vertx.kafka.client.consumer.impl.KafkaConsumerImpl.poll(KafkaConsumerImpl.java:683)
	at io.vertx.kafka.client.consumer.impl.KafkaConsumerImpl.poll(KafkaConsumerImpl.java:695)
	at com.kodypay.lychee.kafka.KafkaConsumer$3.invokeSuspend(KafkaConsumer.kt:65)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
	at kotlinx.coroutines.EventLoop.processUnconfinedEvent(EventLoop.common.kt:65)
	at kotlinx.coroutines.internal.DispatchedContinuationKt.resumeCancellableWith(DispatchedContinuation.kt:371)
	at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable(Cancellable.kt:26)
	at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable$default(Cancellable.kt:21)
	at kotlinx.coroutines.CoroutineStart.invoke(CoroutineStart.kt:88)
	at kotlinx.coroutines.AbstractCoroutine.start(AbstractCoroutine.kt:123)
	at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch(Builders.common.kt:52)
	at kotlinx.coroutines.BuildersKt.launch(Unknown Source)
	at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch$default(Builders.common.kt:43)
	at kotlinx.coroutines.BuildersKt.launch$default(Unknown Source)
	at io.vertx.kotlin.coroutines.CoroutineVerticle.start(CoroutineVerticle.kt:51)
	at io.vertx.core.impl.DeploymentManager.lambda$doDeploy$5(DeploymentManager.java:210)
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:279)
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:261)
	at io.vertx.core.impl.ContextInternal.lambda$runOnContext$0(ContextInternal.java:59)
	at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
	at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:1583)

sishbi avatar Mar 27 '24 18:03 sishbi

And another:

Note: this error occurred about 3 minutes after the k8s pod started. So it is likely the first time the consumer joined the kafka cluster.

Thread Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 2018 ms, time limit is 2000 ms

io.vertx.core.VertxException: Thread blocked
	at java.base/java.security.AccessController.getStackAccessControlContext(Native Method)
	at java.base/java.security.AccessController.getContext(AccessController.java:1006)
	at java.base/java.lang.Thread.<init>(Thread.java:741)
	at java.base/java.lang.Thread.<init>(Thread.java:1275)
	at io.vertx.kafka.client.consumer.impl.KafkaReadStreamImpl.lambda$start$0(KafkaReadStreamImpl.java:115)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:637)
	at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:928)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1364)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:123)
	at java.base/java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:785)
	at io.vertx.kafka.client.consumer.impl.KafkaReadStreamImpl.submitTaskWhenStarted(KafkaReadStreamImpl.java:123)
	at io.vertx.kafka.client.consumer.impl.KafkaReadStreamImpl.start(KafkaReadStreamImpl.java:116)
	at io.vertx.kafka.client.consumer.impl.KafkaReadStreamImpl.subscribe(KafkaReadStreamImpl.java:443)
	at io.vertx.kafka.client.consumer.impl.KafkaConsumerImpl.subscribe(KafkaConsumerImpl.java:197)
	at io.vertx.kafka.client.consumer.impl.KafkaConsumerImpl.subscribe(KafkaConsumerImpl.java:186)
	at io.vertx.kafka.client.consumer.impl.KafkaConsumerImpl.subscribe(KafkaConsumerImpl.java:180)
	at com.kodypay.lychee.kafka.KafkaConsumer.<init>(KafkaConsumer.kt:57)
	at com.kodypay.cards.kafka.payments.CombinedPaymentEventConsumer.<init>(CombinedPaymentEventConsumer.kt:26)
	at com.kodypay.cards.kafka.KafkaVerticle.start(KafkaVerticle.kt:43)
	at io.vertx.kotlin.coroutines.CoroutineVerticle$start$1.invokeSuspend(CoroutineVerticle.kt:53)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.internal.DispatchedContinuationKt.resumeCancellableWith(DispatchedContinuation.kt:363)
	at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable(Cancellable.kt:26)
	at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable$default(Cancellable.kt:21)
	at kotlinx.coroutines.CoroutineStart.invoke(CoroutineStart.kt:88)
	at kotlinx.coroutines.AbstractCoroutine.start(AbstractCoroutine.kt:123)
	at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch(Builders.common.kt:52)
	at kotlinx.coroutines.BuildersKt.launch(Unknown Source)
	at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch$default(Builders.common.kt:43)
	at kotlinx.coroutines.BuildersKt.launch$default(Unknown Source)
	at io.vertx.kotlin.coroutines.CoroutineVerticle.start(CoroutineVerticle.kt:51)
	at io.vertx.core.impl.DeploymentManager.lambda$doDeploy$5(DeploymentManager.java:210)
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:279)
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:261)
	at io.vertx.core.impl.ContextInternal.lambda$runOnContext$0(ContextInternal.java:59)
	at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
	at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:1583)

sishbi avatar Mar 28 '24 08:03 sishbi