kop
kop copied to clipboard
streamnative
pulsar-kop
12:22:09.801 [bookkeeper-ml-workers-OrderedExecutor-0-0] ERROR io.streamnative.pulsar.handlers.kop.format.PulsarEntryFormatter - Meet exception: {} java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Maximum offset delta exceeded, base offset: 475692710641008640, last offset: 503769839567896576 at sun.reflect.GeneratedConstructorAccessor103.newInstance(Unknown Source) ~[?:?] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_211] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_211] at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:593) ~[?:1.8.0_211] at java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:677) ~[?:1.8.0_211] at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:735) ~[?:1.8.0_211] at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:158) ~[?:1.8.0_211] at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:174) ~[?:1.8.0_211] at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) ~[?:1.8.0_211] at java.util.stream.ReferencePipeline.forEachOrdered(ReferencePipeline.java:423) ~[?:1.8.0_211] at java.util.stream.ReferencePipeline$Head.forEachOrdered(ReferencePipeline.java:593) ~[?:1.8.0_211] at io.streamnative.pulsar.handlers.kop.format.PulsarEntryFormatter.decode(PulsarEntryFormatter.java:140) ~[?:?] at io.streamnative.pulsar.handlers.kop.MessageFetchContext.lambda$null$11(MessageFetchContext.java:324) ~[?:?] at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) ~[?:1.8.0_211] at java.util.concurrent.ConcurrentHashMap$EntrySpliterator.forEachRemaining(ConcurrentHashMap.java:3606) ~[?:1.8.0_211] at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_211] at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291) ~[?:1.8.0_211] at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) ~[?:1.8.0_211] at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) ~[?:1.8.0_211]
Add following config to your conf/broker.conf:
maxReadEntriesNum=1
Or you can wait for the incoming KoP 2.8.0 that may be released in one to two weeks.
@Hanfee Hi, now KoP 2.8.0 is released, see https://github.com/streamnative/kop/releases/tag/v2.8.0.1. This problem won't happen anymore. For old versions KoP, have you tried with my suggestions?
I use PulsarIO KafkaConnector replace Kop temporarily ，Now I try update kop to 2.8.0