micronaut-core
micronaut-core copied to clipboard
micronaut-projects
Potential Memory Leak
Issue description
A user upgrading from 3.9.2 to 4 pointed to a potential memory leak
Our application (running on micronaut-platform 4.4.5) is crashing with an out of memory error.
After some analysis of the heap dump, we've found that the issue is due to DirectByteBuffers not being released properly.
We've then enabled this flag to track netty byte buffers:
-Dio.netty.leakDetectionLevel=paranoid
The logs seems to report that the error happens while writing a json response in the micronaut http server:
2024-07-17 13:15:27.048 Jul-17 11:15:27.046 [default-nioEventLoopGroup-5-9] 86.144.114.222 ERROR io.netty.util.ResourceLeakDetector - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
#1:
io.netty.buffer.AdvancedLeakAwareByteBuf.writeBytes(AdvancedLeakAwareByteBuf.java:617)
io.netty.buffer.ByteBufOutputStream.write(ByteBufOutputStream.java:80)
com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2210)
com.fasterxml.jackson.core.json.UTF8JsonGenerator.close(UTF8JsonGenerator.java:1234)
com.fasterxml.jackson.databind.ObjectMapper._writeValueAndClose(ObjectMapper.java:4804)
com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:3999)
io.micronaut.jackson.databind.JacksonDatabindMapper.writeValue(JacksonDatabindMapper.java:207)
io.micronaut.http.netty.body.NettyJsonHandler.writeTo(NettyJsonHandler.java:161)
io.micronaut.http.server.netty.RoutingInBoundHandler.writeNettyMessageBody(RoutingInBoundHandler.java:377)
io.micronaut.http.server.netty.RoutingInBoundHandler.encodeHttpResponse(RoutingInBoundHandler.java:355)
io.micronaut.http.server.netty.RoutingInBoundHandler.writeResponse(RoutingInBoundHandler.java:249)
io.micronaut.http.server.netty.NettyRequestLifecycle.lambda$handleNormal$0(NettyRequestLifecycle.java:94)
io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl$1.onComplete(ReactorExecutionFlowImpl.java:121)
reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:159)
reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
io.micronaut.configuration.metrics.binder.web.WebMetricsPublisher$1.onNext(WebMetricsPublisher.java:178)
io.micronaut.configuration.metrics.binder.web.WebMetricsPublisher$1.onNext(WebMetricsPublisher.java:163)
reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
reactor.core.publisher.MonoFlatMap$FlatMapMain.secondComplete(MonoFlatMap.java:245)
reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:305)
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onNext(MonoFlatMapMany.java:251)
reactor.core.publisher.Operators$MonoInnerProducerBase.complete(Operators.java:2812)
reactor.core.publisher.SinkOneMulticast.tryEmitValue(SinkOneMulticast.java:66)
reactor.core.publisher.SinkOneSerialized.tryEmitValue(SinkOneSerialized.java:38)
io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl$2.accept(ReactorExecutionFlowImpl.java:171)
io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl$2.accept(ReactorExecutionFlowImpl.java:164)
io.micronaut.core.execution.ImperativeExecutionFlowImpl.onComplete(ImperativeExecutionFlowImpl.java:132)
io.micronaut.core.execution.DelayedExecutionFlowImpl$OnComplete.apply(DelayedExecutionFlowImpl.java:330)
io.micronaut.core.execution.DelayedExecutionFlowImpl.work(DelayedExecutionFlowImpl.java:51)
io.micronaut.core.execution.DelayedExecutionFlowImpl.complete0(DelayedExecutionFlowImpl.java:64)
io.micronaut.core.execution.DelayedExecutionFlowImpl.complete(DelayedExecutionFlowImpl.java:70)
io.micronaut.core.execution.ExecutionFlow.lambda$async$0(ExecutionFlow.java:94)
io.micronaut.core.execution.ImperativeExecutionFlowImpl.onComplete(ImperativeExecutionFlowImpl.java:132)
io.micronaut.core.execution.ExecutionFlow.lambda$async$1(ExecutionFlow.java:87)
io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:141)
io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:193)
io.micronaut.core.propagation.PropagatedContext.lambda$wrap$3(PropagatedContext.java:211)
io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:141)
io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:193)
io.micronaut.core.propagation.PropagatedContext.lambda$wrap$3(PropagatedContext.java:211)
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
java.base/java.lang.Thread.run(Thread.java:840)
Created at:
io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:404)
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:174)
io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:108)
io.micronaut.http.netty.body.NettyJsonHandler.writeTo(NettyJsonHandler.java:158)
io.micronaut.http.server.netty.RoutingInBoundHandler.writeNettyMessageBody(RoutingInBoundHandler.java:377)
io.micronaut.http.server.netty.RoutingInBoundHandler.encodeHttpResponse(RoutingInBoundHandler.java:355)
io.micronaut.http.server.netty.RoutingInBoundHandler.writeResponse(RoutingInBoundHandler.java:249)
io.micronaut.http.server.netty.NettyRequestLifecycle.lambda$handleNormal$0(NettyRequestLifecycle.java:94)
io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl$1.onComplete(ReactorExecutionFlowImpl.java:121)
reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:159)
reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
io.micronaut.configuration.metrics.binder.web.WebMetricsPublisher$1.onNext(WebMetricsPublisher.java:178)
io.micronaut.configuration.metrics.binder.web.WebMetricsPublisher$1.onNext(WebMetricsPublisher.java:163)
reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
reactor.core.publisher.MonoFlatMap$FlatMapMain.secondComplete(MonoFlatMap.java:245)
reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:305)
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onNext(MonoFlatMapMany.java:251)
reactor.core.publisher.Operators$MonoInnerProducerBase.complete(Operators.java:2812)
reactor.core.publisher.SinkOneMulticast.tryEmitValue(SinkOneMulticast.java:66)
reactor.core.publisher.SinkOneSerialized.tryEmitValue(SinkOneSerialized.java:38)
io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl$2.accept(ReactorExecutionFlowImpl.java:171)
io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl$2.accept(ReactorExecutionFlowImpl.java:164)
io.micronaut.core.execution.ImperativeExecutionFlowImpl.onComplete(ImperativeExecutionFlowImpl.java:132)
io.micronaut.core.execution.DelayedExecutionFlowImpl$OnComplete.apply(DelayedExecutionFlowImpl.java:330)
io.micronaut.core.execution.DelayedExecutionFlowImpl.work(DelayedExecutionFlowImpl.java:51)
io.micronaut.core.execution.DelayedExecutionFlowImpl.complete0(DelayedExecutionFlowImpl.java:64)
io.micronaut.core.execution.DelayedExecutionFlowImpl.complete(DelayedExecutionFlowImpl.java:70)
io.micronaut.core.execution.ExecutionFlow.lambda$async$0(ExecutionFlow.java:94)
io.micronaut.core.execution.ImperativeExecutionFlowImpl.onComplete(ImperativeExecutionFlowImpl.java:132)
io.micronaut.core.execution.ExecutionFlow.lambda$async$1(ExecutionFlow.java:87)
io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:141)
io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:193)
We've tried to update to the latest available micronaut version 4.5.0, to check if the issue was solved already, and run the application in a separate environment, but the leak is still detected in the same place.
2024-07-15 15:58:23.630 Jul-15 13:58:23.629 [io-executor-thread-6] 82.61.68.148 ERROR io.netty.util.ResourceLeakDetector - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
#1:
io.netty.buffer.AdvancedLeakAwareByteBuf.writeBytes(AdvancedLeakAwareByteBuf.java:617)
io.netty.buffer.ByteBufOutputStream.write(ByteBufOutputStream.java:80)
com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2210)
com.fasterxml.jackson.core.json.UTF8JsonGenerator.close(UTF8JsonGenerator.java:1234)
com.fasterxml.jackson.databind.ObjectMapper._writeValueAndClose(ObjectMapper.java:4804)
com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:3999)
io.micronaut.jackson.databind.JacksonDatabindMapper.writeValue(JacksonDatabindMapper.java:207)
io.micronaut.http.netty.body.NettyJsonHandler.writeTo(NettyJsonHandler.java:163)
io.micronaut.http.server.netty.RoutingInBoundHandler.writeNettyMessageBody(RoutingInBoundHandler.java:380)
io.micronaut.http.server.netty.RoutingInBoundHandler.encodeHttpResponse(RoutingInBoundHandler.java:358)
io.micronaut.http.server.netty.RoutingInBoundHandler.writeResponse(RoutingInBoundHandler.java:248)
io.micronaut.http.server.netty.NettyRequestLifecycle.lambda$handleNormal$0(NettyRequestLifecycle.java:101)
io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl$1.onComplete(ReactorExecutionFlowImpl.java:121)
reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:159)
reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
io.micronaut.configuration.metrics.binder.web.WebMetricsPublisher$1.onNext(WebMetricsPublisher.java:178)
io.micronaut.configuration.metrics.binder.web.WebMetricsPublisher$1.onNext(WebMetricsPublisher.java:163)
reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
reactor.core.publisher.MonoFlatMap$FlatMapMain.secondComplete(MonoFlatMap.java:245)
reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:305)
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onNext(MonoFlatMapMany.java:251)
reactor.core.publisher.Operators$MonoInnerProducerBase.complete(Operators.java:2834)
reactor.core.publisher.SinkOneMulticast.tryEmitValue(SinkOneMulticast.java:66)
reactor.core.publisher.SinkOneSerialized.tryEmitValue(SinkOneSerialized.java:38)
io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl$2.accept(ReactorExecutionFlowImpl.java:171)
io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl$2.accept(ReactorExecutionFlowImpl.java:164)
io.micronaut.core.execution.ImperativeExecutionFlowImpl.onComplete(ImperativeExecutionFlowImpl.java:132)
io.micronaut.core.execution.DelayedExecutionFlowImpl$OnComplete.apply(DelayedExecutionFlowImpl.java:330)
io.micronaut.core.execution.DelayedExecutionFlowImpl.work(DelayedExecutionFlowImpl.java:51)
io.micronaut.core.execution.DelayedExecutionFlowImpl.complete0(DelayedExecutionFlowImpl.java:64)
io.micronaut.core.execution.DelayedExecutionFlowImpl.complete(DelayedExecutionFlowImpl.java:70)
io.micronaut.core.execution.ExecutionFlow.lambda$async$0(ExecutionFlow.java:94)
io.micronaut.core.execution.ImperativeExecutionFlowImpl.onComplete(ImperativeExecutionFlowImpl.java:132)
io.micronaut.core.execution.ExecutionFlow.lambda$async$1(ExecutionFlow.java:87)
io.micronaut.core.propagation.PropagatedContext.lambda$wrap$3(PropagatedContext.java:211)
io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:141)
io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:193)
io.micronaut.core.propagation.PropagatedContext.lambda$wrap$3(PropagatedContext.java:211)
io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:141)
io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:193)
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
java.base/java.lang.Thread.run(Thread.java:840)
Created at:
io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:404)
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:174)
io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:108)
io.micronaut.http.netty.body.NettyJsonHandler.writeTo(NettyJsonHandler.java:160)
io.micronaut.http.server.netty.RoutingInBoundHandler.writeNettyMessageBody(RoutingInBoundHandler.java:380)
io.micronaut.http.server.netty.RoutingInBoundHandler.encodeHttpResponse(RoutingInBoundHandler.java:358)
io.micronaut.http.server.netty.RoutingInBoundHandler.writeResponse(RoutingInBoundHandler.java:248)
io.micronaut.http.server.netty.NettyRequestLifecycle.lambda$handleNormal$0(NettyRequestLifecycle.java:101)
io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl$1.onComplete(ReactorExecutionFlowImpl.java:121)
reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:159)
reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
io.micronaut.configuration.metrics.binder.web.WebMetricsPublisher$1.onNext(WebMetricsPublisher.java:178)
io.micronaut.configuration.metrics.binder.web.WebMetricsPublisher$1.onNext(WebMetricsPublisher.java:163)
reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
io.micronaut.core.async.propagation.ReactivePropagation$2.onNext(ReactivePropagation.java:106)
reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
reactor.core.publisher.MonoFlatMap$FlatMapMain.secondComplete(MonoFlatMap.java:245)
reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:305)
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onNext(MonoFlatMapMany.java:251)
reactor.core.publisher.Operators$MonoInnerProducerBase.complete(Operators.java:2834)
reactor.core.publisher.SinkOneMulticast.tryEmitValue(SinkOneMulticast.java:66)
reactor.core.publisher.SinkOneSerialized.tryEmitValue(SinkOneSerialized.java:38)
io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl$2.accept(ReactorExecutionFlowImpl.java:171)
io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl$2.accept(ReactorExecutionFlowImpl.java:164)
io.micronaut.core.execution.ImperativeExecutionFlowImpl.onComplete(ImperativeExecutionFlowImpl.java:132)
io.micronaut.core.execution.DelayedExecutionFlowImpl$OnComplete.apply(DelayedExecutionFlowImpl.java:330)
io.micronaut.core.execution.DelayedExecutionFlowImpl.work(DelayedExecutionFlowImpl.java:51)
io.micronaut.core.execution.DelayedExecutionFlowImpl.complete0(DelayedExecutionFlowImpl.java:64)
io.micronaut.core.execution.DelayedExecutionFlowImpl.complete(DelayedExecutionFlowImpl.java:70)
io.micronaut.core.execution.ExecutionFlow.lambda$async$0(ExecutionFlow.java:94)
io.micronaut.core.execution.ImperativeExecutionFlowImpl.onComplete(ImperativeExecutionFlowImpl.java:132)
io.micronaut.core.execution.ExecutionFlow.lambda$async$1(ExecutionFlow.java:87)
io.micronaut.core.propagation.PropagatedContext.lambda$wrap$3(PropagatedContext.java:211)
io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:141)
The same request doesn't always produce the log, but we see a steady increase in the size of the memory used by the application.
We are using jackson-databind for serialization and we cannot move to the new micronaut serialization approach just yet.
Inspecting the micronaut codebase, we've found that the buffer allocated in https://github.com/micronaut-projects/micronaut-core/blob/df3a78fd20091e6a773369fc0e942bb51d806b76/http-netty/src/main/java/io/micronaut/http/netty/body/NettyJsonHandler.java#L160 is not released immediately but this is delegated to the DefaultFullHttpResponse#release() method but we where not able to track it further.
@yawkat when creating the ByteBufOutputStream, we are not using the constructor which releases on close . I assume that it is intentional because we pass the ByteBuff to DefaultFullHttpResponse and we expect someone to invoke DefaultFullHttpResponse::release. Who is doing that?
Yes it's intentional. PipeliningServerHandler does the release. This is not sufficient information to determine the cause of the leak.
Any update on this, facing the similar issue while running load test on micronaut application. Have written POST endpoint which accepts InputStream
Detailed stack trace of error: {"timestamp":"2024-11-14 16:37:42,247","logger":"io.netty.util.ResourceLeakDetector","level":"ERROR","thread":"default-nioEventLoopGroup-1-20","message":" was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.\nRecent access records: \n#1:\n\tio.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:300)\n\tio.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tio.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tio.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tio.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tio.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1407)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tio.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918)\n\tio.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)\n\tio.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)\n\tio.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)\n\tio.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)\n\tio.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)\n\tio.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)\n\tio.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tio.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tjava.base/java.lang.Thread.run(Thread.java:840)\n#2:\n\tio.netty.buffer.AdvancedLeakAwareByteBuf.getByte(AdvancedLeakAwareByteBuf.java:155)\n\tio.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:1160)\n\tio.netty.handler.codec.http.HttpObjectDecoder$LineParser.parse(HttpObjectDecoder.java:1210)\n\tio.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:456)\n\tio.netty.handler.codec.http.HttpServerCodec$HttpServerRequestDecoder.decode(HttpServerCodec.java:167)\n\tio.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)\n\tio.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)\n\tio.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)\n\tio.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tio.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tio.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tio.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tio.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1407)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tio.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918)\n\tio.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)\n\tio.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)\n\tio.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)\n\tio.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)\n\tio.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)\n\tio.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)\n\tio.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tio.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tjava.base/java.lang.Thread.run(Thread.java:840)\n#3:\n\tio.netty.buffer.AdvancedLeakAwareByteBuf.readRetainedSlice(AdvancedLeakAwareByteBuf.java:107)\n\tio.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:482)\n\tio.netty.handler.codec.http.HttpServerCodec$HttpServerRequestDecoder.decode(HttpServerCodec.java:167)\n\tio.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)\n\tio.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)\n\tio.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)\n\tio.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tio.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tio.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tio.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tio.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1407)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tio.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918)\n\tio.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)\n\tio.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)\n\tio.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)\n\tio.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)\n\tio.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)\n\tio.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)\n\tio.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tio.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tjava.base/java.lang.Thread.run(Thread.java:840)\n#4:\n\tio.netty.buffer.AdvancedLeakAwareByteBuf.getByte(AdvancedLeakAwareByteBuf.java:155)\n\tio.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:497)\n\tio.netty.handler.codec.http.HttpServerCodec$HttpServerRequestDecoder.decode(HttpServerCodec.java:167)\n\tio.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)\n\tio.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)\n\tio.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)\n\tio.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tio.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tio.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tio.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tio.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1407)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tio.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918)\n\tio.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)\n\tio.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)\n\tio.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)\n\tio.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)\n\tio.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)\n\tio.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)\n\tio.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tio.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tjava.base/java.lang.Thread.run(Thread.java:840)\n#5:\n\tHint: 'http-server-codec' will handle the message from this point.\n\tio.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:115)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:417)\n\tio.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tio.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tio.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tio.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1407)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tio.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918)\n\tio.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)\n\tio.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)\n\tio.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)\n\tio.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)\n\tio.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)\n\tio.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)\n\tio.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tio.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tjava.base/java.lang.Thread.run(Thread.java:840)\n#6:\n\tHint: 'idle-state' will handle the message from this point.\n\tio.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:115)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:417)\n\tio.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tio.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1407)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tio.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918)\n\tio.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)\n\tio.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)\n\tio.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)\n\tio.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)\n\tio.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)\n\tio.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)\n\tio.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tio.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tjava.base/java.lang.Thread.run(Thread.java:840)\n#7:\n\tHint: 'DefaultChannelPipeline$HeadContext#0' will handle the message from this point.\n\tio.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:115)\n\tio.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:417)\n\tio.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918)\n\tio.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)\n\tio.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)\n\tio.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)\n\tio.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)\n\tio.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)\n\tio.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)\n\tio.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tio.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tjava.base/java.lang.Thread.run(Thread.java:840)\n#8:\n\tio.netty.buffer.AdvancedLeakAwareByteBuf.writeBytes(AdvancedLeakAwareByteBuf.java:635)\n\tio.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:356)\n\tio.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)\n\tio.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)\n\tio.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)\n\tio.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)\n\tio.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)\n\tio.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)\n\tio.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tio.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tjava.base/java.lang.Thread.run(Thread.java:840)\nCreated at:\n\tio.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:404)\n\tio.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)\n\tio.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)\n\tio.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:140)\n\tio.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:120)\n\tio.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:150)\n\tio.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)\n\tio.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)\n\tio.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)\n\tio.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)\n\tio.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)\n\tio.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tio.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tjava.base/java.lang.Thread.run(Thread.java:840)\n: 8 leak records were discarded because the leak record count is targeted to 4. Use system property io.netty.leakDetection.targetRecords to increase the limit."}
This is post request for the application written using Micronaut core vserions : 4.7.5
curl --location 'http://localhost:8080/cloudstore/data/customer/test1'
--header 'Content-Type: application/octet-stream'
--data-binary '@postman-cloud:///1ef5a2f1-65e0-4d40-b0df-c81ae3f06952'
If I run load test for this curl, say 80 request per second. I can see direct byteBuffer memory increasing and not coming down even after stopping application.
Enabling io.netty.leak.Detection.level=paranoid gives the stack trace which I have pasted earlier. In the same request if I change the content-type to text/plain. Problem goes away
@deepi2003 I created a simple application, with the code below:
@Controller("/upload")
public class OcController {
@Post(consumes = "application/octet-stream")
public HttpResponse<String> upload(@Body byte[] fileData) throws IOException {
System.out.println("echo");
return HttpResponse.ok("File uploaded!");
}
}
I sent 500k requests and in the heap dump I was unable to detect any memory leak, if it was something that generally affected the octet-stream the example would probably be affected, can you send an example where the problem occurs?
