Get exception when using micrometer

[richiesgr]

Hi when I switch using com.pinterest.secor.monitoring.MicroMeterMetricCollector instead of com.pinterest.secor.monitoring.OstrichMetricCollector I get thousands of exception WARN Error while discarding collection, stopping java.util.ConcurrentModificationException at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909) at java.util.ArrayList$Itr.next(ArrayList.java:859) at io.micrometer.shaded.reactor.core.publisher.Operators.onDiscardMultiple(Operators.java:565) at io.micrometer.shaded.reactor.core.publisher.FluxBufferPredicate$BufferPredicateSubscriber.cancel(FluxBufferPredicate.java:174) at io.micrometer.shaded.reactor.core.publisher.FluxMap$MapSubscriber.cancel(FluxMap.java:160) at io.micrometer.shaded.reactor.core.publisher.FluxOnBackpressureLatest$LatestSubscriber.cancel(FluxOnBackpressureLatest.java:99) at io.micrometer.shaded.reactor.core.publisher.FluxMap$MapSubscriber.cancel(FluxMap.java:160) at io.micrometer.shaded.reactor.core.publisher.Operators.terminate(Operators.java:1222) at io.micrometer.shaded.reactor.netty.channel.MonoSendMany$SendManyInner.tryFailure(MonoSendMany.java:577) at io.micrometer.shaded.io.netty.util.internal.PromiseNotificationUtil.tryFailure(PromiseNotificationUtil.java:64) at io.micrometer.shaded.io.netty.channel.ChannelOutboundBuffer.safeFail(ChannelOutboundBuffer.java:723) at io.micrometer.shaded.io.netty.channel.ChannelOutboundBuffer.remove0(ChannelOutboundBuffer.java:308) at io.micrometer.shaded.io.netty.channel.ChannelOutboundBuffer.remove(ChannelOutboundBuffer.java:288) at io.micrometer.shaded.io.netty.channel.nio.AbstractNioMessageChannel.doWrite(AbstractNioMessageChannel.java:159) at io.micrometer.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:930) at io.micrometer.shaded.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:354) at io.micrometer.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:897) at io.micrometer.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1372) at io.micrometer.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750) at io.micrometer.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:742) at io.micrometer.shaded.io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:728) at io.micrometer.shaded.reactor.netty.channel.MonoSendMany$SendManyInner$AsyncFlush.run(MonoSendMany.java:758) at io.micrometer.shaded.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at io.micrometer.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at io.micrometer.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at io.micrometer.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.micrometer.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.micrometer.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

my config are secor.monitoring.metrics.collector.micrometer.jmx.enabled=true secor.monitoring.metrics.collector.micrometer.statsd.enabled=true secor.monitoring.metrics.collector.micrometer.prometheus.enabled=true

[pdambrauskas]

Looks like https://github.com/micrometer-metrics/micrometer/issues/2238

It is a bug in micrometer version < 1.5.5. Latest secor release uses 1.5.2. You can try to fix this issue by building secor from master (dependencies are updated there). Or disabling statsd and waiting for new release.

[HenryCaiHaiying]

The latest secor is using micrometer 1.6.1

On Tue, Dec 1, 2020 at 5:44 AM Paulius [email protected] wrote:

Looks like micrometer-metrics/micrometer#2238 https://github.com/micrometer-metrics/micrometer/issues/2238

It is a bug in micrometer version < 1.5.5. Latest secor release uses 1.5.2. You can try to fix this issue by building secor from master (dependencies are updated there). Or disabling statsd and waiting for new release.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/pinterest/secor/issues/1732#issuecomment-736560042, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABYJP73RPVKJKKEQWHY7PDDSSTXLXANCNFSM4UJAQLSA .

[pdambrauskas]

The latest secor is using micrometer 1.6.1

But release tagged with 0.29 uses 1.5.2 :) https://github.com/pinterest/secor/blob/v0.29/pom.xml#L381

[HenryCaiHaiying]

We can create a new tag as well if the code at head fixes the problem.

On Wed, Dec 2, 2020 at 12:45 AM Paulius [email protected] wrote:

The latest secor is using micrometer 1.6.1

But release tagged with 0.29 uses 1.5.2 :) https://github.com/pinterest/secor/blob/v0.29/pom.xml#L381

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/pinterest/secor/issues/1732#issuecomment-737082902, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABYJP72KYHGBXQLWFO3SFN3SSX5BJANCNFSM4UJAQLSA .