Collector stops working on Windows server 2k12 with exception

[gruselglatz]

Hi,

Our Collector-File-module 0.4.2 stops working ever night with this exception:

Exception in thread "ChunkProcessor" java.lang.IndexOutOfBoundsException: readerIndex(0) + length(174) exceeds writerIndex(160): CompositeByteBuf(ridx: 0, widx: 160, cap: 160, components=1)
        at io.netty.buffer.AbstractByteBuf.checkReadableBytes(AbstractByteBuf.java:1166)
        at io.netty.buffer.AbstractByteBuf.skipBytes(AbstractByteBuf.java:741)
        at io.netty.buffer.CompositeByteBuf.skipBytes(CompositeByteBuf.java:1511)
        at io.netty.buffer.CompositeByteBuf.skipBytes(CompositeByteBuf.java:42)
        at org.graylog.collector.file.splitters.PatternChunkSplitter$1$1.computeNext(PatternChunkSplitter.java:75)
        at org.graylog.collector.file.splitters.PatternChunkSplitter$1$1.computeNext(PatternChunkSplitter.java:40)
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
        at org.graylog.collector.file.ChunkProcessor.createMessages(ChunkProcessor.java:89)
        at org.graylog.collector.file.ChunkProcessor.process(ChunkProcessor.java:85)
        at org.graylog.collector.file.ChunkProcessor.run(ChunkProcessor.java:51)
        at com.google.common.util.concurrent.AbstractExecutionThreadService$1$2.run(AbstractExecutionThreadService.java:60)
        at com.google.common.util.concurrent.Callables$3.run(Callables.java:95)
        at java.lang.Thread.run(Thread.java:745)

The Windows Event-module is still sending. After this exception it is also not possible to stop the service without killing the process!

For the Explanation this file contains standard Windows-Server DNS Messages.

thx

[wentinkj]

Hi,

We're experiencing the same problem with 0.4.0 and jre 1.7.0_80 and windows server 2k12.. We're also using a pattern based splitter. By explicitly adding charset = "utf-8" to the input and client-send-buffer-size = 32768 to the output the collector will keep running (longer). Only the busiest environments still die after a bit more than a day. In our situation the joined lines can become quite large, but there doesn't seem to be a direct relationship to that. It's also counter intuitive to what's described here - bytebuf-. Where it states that this exception is thrown when there's not enough to read.

Anyone any clues on what to try next?

[gruselglatz]

Hi wentinkj,

we've began to restart the collector every 15 minutes with a task. I'ts nod ideal but until there is an update we are good with that solution.

[chrkspr]

Hello everyone,

I have the same problem with ibm jre 1.7 (64bit) und 1.8 (32bit) on linux (rhel 6.7, 64bit) with graylog-collector 0.4.2. I am using a pattern based splitter as well.

Best, Christian

[msteelepmp]

I'm also seeing this with collector 0.4.2 on CentOS 6.7, Oracle java version "1.8.0_91"

2016-04-21T19:31:51.571+0000 INFO  [main] cli.commands.Run - Service RUNNING: FileInput{outputs='', content-splitter='NEWLINE', charset='UTF-8', message-fields='MessageFields{}', reader-buffer-size='102400', reader-interval='100', id='log-files', path-set='GlobPathSet{rootPath=/var/log, pattern=**/*.{log,txt}}'}
2016-04-21T19:31:51.580+0000 INFO  [main] cli.commands.Run - Service RUNNING: GelfOutput{client-send-buffer-size='-1', port='12201', inputs='', host='graylog.librasys.com', client-reconnect-delay='1000', client-tcp-no-delay='true', id='gelf-tcp', client-queue-size='512', client-connect-timeout='5000'}
Exception in thread "ChunkProcessor" java.lang.IndexOutOfBoundsException: readerIndex(28) + length(1) exceeds writerIndex(28): CompositeByteBuf(ridx: 28, widx: 28, cap: 28, components=1)
    at io.netty.buffer.AbstractByteBuf.checkReadableBytes(AbstractByteBuf.java:1166)
    at io.netty.buffer.AbstractByteBuf.readByte(AbstractByteBuf.java:570)
    at org.graylog.collector.file.splitters.NewlineChunkSplitter$1$1.computeNext(NewlineChunkSplitter.java:53)
    at org.graylog.collector.file.splitters.NewlineChunkSplitter$1$1.computeNext(NewlineChunkSplitter.java:32)
    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
    at org.graylog.collector.file.ChunkProcessor.createMessages(ChunkProcessor.java:89)
    at org.graylog.collector.file.ChunkProcessor.process(ChunkProcessor.java:85)
    at org.graylog.collector.file.ChunkProcessor.run(ChunkProcessor.java:51)
    at com.google.common.util.concurrent.AbstractExecutionThreadService$1$2.run(AbstractExecutionThreadService.java:60)
    at com.google.common.util.concurrent.Callables$3.run(Callables.java:95)
    at java.lang.Thread.run(Thread.java:745)