Kinesis agent throwing illegal state exception after log rotation

[SripadaBhargav]

In our application, kinesis agent is trying to read a file abc.log, at the same time, abc.log is rotated using logrotate with copytruncate mode. After the file got rotated to sub folder, kinesis agent could not parse the content in the file and starts throwing java.lang.IllegalStateException continuously.

Log rotate config /var/log/tomcat7/* { size 10M rotate 5 missingok compress notifempty copytruncate dateext dateformat %s olddir /var/log/tomcat7/rotated }

Exception:

Error when processing current input file or when tracking its status. java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:158) at com.amazon.kinesis.streaming.agent.tailing.TrackedFileRotationAnalyzer.findCurrentOpenFileAfterTruncate(Unknown Source) at com.amazon.kinesis.streaming.agent.tailing.SourceFileTracker.updateCurrentFile(Unknown Source) at com.amazon.kinesis.streaming.agent.tailing.SourceFileTracker.refresh(Unknown Source) at com.amazon.kinesis.streaming.agent.tailing.FileTailer.updateRecordParser(Unknown Source) at com.amazon.kinesis.streaming.agent.tailing.FileTailer.processRecords(Unknown Source) at com.amazon.kinesis.streaming.agent.tailing.FileTailer.runOnce(Unknown Source) at com.amazon.kinesis.streaming.agent.tailing.FileTailer.run(Unknown Source) at com.google.common.util.concurrent.AbstractExecutionThreadService$1$2.run(AbstractExecutionThreadService.java:60) at com.google.common.util.concurrent.Callables$3.run(Callables.java:95) at java.lang.Thread.run(Thread.java:748)

[segwitrip]

Looks like a dupe of 109

[kimyoon21]

is there no comment about it?

[Gabriellpweb]

Something similar is happening with me too, every rotate the kinesis go crazy and i need to remove his checkpoints. I'm almost moving out to fluentd, kinesis agents seems broken.

Initialization logs can be found in /tmp/aws-kinesis-agent.20190712081053.initlog find: failed to restore initial working directory: Permission denied Unhandled exception during startup. java.lang.IllegalStateException: Expected the service to be RUNNING, but the service has FAILED at com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:285) at com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:229) at com.google.common.util.concurrent.AbstractExecutionThreadService.awaitRunning(AbstractExecutionThreadService.java:197) at com.amazon.kinesis.streaming.agent.Agent.startUp(Unknown Source) at com.google.common.util.concurrent.AbstractIdleService$2$1.run(AbstractIdleService.java:54) at com.google.common.util.concurrent.Callables$3.run(Callables.java:95) at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) at com.google.common.util.concurrent.MoreExecutors$2.execute(MoreExecutors.java:849) at com.google.common.util.concurrent.AbstractIdleService$2.doStart(AbstractIdleService.java:50) at com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:174) at com.google.common.util.concurrent.AbstractIdleService.startAsync(AbstractIdleService.java:135) at com.amazon.kinesis.streaming.agent.Agent.main(Unknown Source) Caused by: java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:158) at com.amazon.kinesis.streaming.agent.tailing.TrackedFileRotationAnalyzer.findCurrentOpenFileAfterTruncate(Unknown Source) at com.amazon.kinesis.streaming.agent.tailing.SourceFileTracker.updateCurrentFile(Unknown Source) at com.amazon.kinesis.streaming.agent.tailing.SourceFileTracker.initialize(Unknown Source) at com.amazon.kinesis.streaming.agent.tailing.FileTailer.initialize(Unknown Source) at com.amazon.kinesis.streaming.agent.tailing.FileTailer.startUp(Unknown Source) at com.google.common.util.concurrent.AbstractExecutionThreadService$1$2.run(AbstractExecutionThreadService.java:55) at com.google.common.util.concurrent.Callables$3.run(Callables.java:95) at java.lang.Thread.run(Thread.java:748) FATAL: Thread FileTailer[fh:app-providers-stream:/var/log/app/requests.log*] threw an unrecoverable error. Aborting application java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:158) at com.amazon.kinesis.streaming.agent.tailing.TrackedFileRotationAnalyzer.findCurrentOpenFileAfterTruncate(Unknown Source) at com.amazon.kinesis.streaming.agent.tailing.SourceFileTracker.updateCurrentFile(Unknown Source) at com.amazon.kinesis.streaming.agent.tailing.SourceFileTracker.initialize(Unknown Source) at com.amazon.kinesis.streaming.agent.tailing.FileTailer.initialize(Unknown Source) at com.amazon.kinesis.streaming.agent.tailing.FileTailer.startUp(Unknown Source) at com.google.common.util.concurrent.AbstractExecutionThreadService$1$2.run(AbstractExecutionThreadService.java:55) at com.google.common.util.concurrent.Callables$3.run(Callables.java:95) at java.lang.Thread.run(Thread.java:748) aws-kinesis-agent startup

/var/log/app/*.log { daily rotate 7 copytruncate compress missingok notifempty postrotate /etc/init.d/aws-kinesis-agent restart > /dev/null 2>/dev/null || true endscript }