smallrye
Mutiny is creating zombie thread on delayIt() which are not doing anything after certain time
We have retry mechanism in our application which waits for some time (2 seconds) and then do the retry.
If we use this in our application throughput is getting dropped to absolute zero after 2 -3 days, thread are going into unresponsive state.
Attached screenshots of our JVM dashboard.
Code block which we have in our mutiny pipeline is as below.
return
Uni.combine().all().unis(delayUni, refreshedCredentialsUni)
.asTuple()
.onItem()
.delayIt()
.by(Duration.ofSeconds(Math.min(backOffDuration, getMaxBackoff())))
.onItem()
.transformToUni(r -> cspResponseUni)
I have found similar thread here --> https://github.com/smallrye/smallrye-mutiny/issues/786
Once we disable this delay, our throughput reduction problem is solved and we are not seeing this issue anymore.
But we need mechanism wherein the above code should work seam lessly.
Are you running in Quarkus? There were a leak in the Jboss scheduler used in Quarkus.
As a workaround, you could define your own executor to execute the delay (instead of using the default one).
@cescoffier Yes, we are running in Quarkus.
Despite supplying an external thread pool we are still observing the same behavior.
engineExecutorService = Executors .newFixedThreadPool(engineThreadPoolSize, new DefaultThreadFactory("worker"));
emitting on below thread pool
.emitOn(EngineThreadPool.getEngineExecutorService());
Ah no, sorry, not using emitOn, but onItem().delayIt().on(executor).by(..)
If you're referring to https://github.com/jbossas/jboss-threads/issues/139 then yes it was fixed.
