KPL cannot restart daemon due to RuntimeException

[johnkchiu]

We noticed our service was getting a lot of these exceptions for each message sent via KPL:

com.amazonaws.services.kinesis.producer.DaemonException: The child process has been shutdown and can no longer accept messages.
	at com.amazonaws.services.kinesis.producer.Daemon.add(Daemon.java:176)
	at com.amazonaws.services.kinesis.producer.KinesisProducer.addUserRecord(KinesisProducer.java:536)
...

Further investigation, we noticed our service logged this exception:

java.lang.IllegalMonitorStateException: null
	at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:151)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1261)
	at java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:457)
	at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1478)
	at com.amazonaws.services.kinesis.producer.Daemon.fatalError(Daemon.java:528)
	at com.amazonaws.services.kinesis.producer.Daemon.fatalError(Daemon.java:513)
	at com.amazonaws.services.kinesis.producer.Daemon.add(Daemon.java:183)
	at com.amazonaws.services.kinesis.producer.KinesisProducer.addUserRecord(KinesisProducer.java:536)
...

We believe the cause is: https://github.com/awslabs/amazon-kinesis-producer/blob/ad22b8bb7ea207498f96f68cff56205daf317c21/java/amazon-kinesis-producer/src/main/java/com/amazonaws/services/kinesis/producer/Daemon.java#L527-L529

1. Daemon.add(message) is interrupted (due to script resource limit reached)
2. Daemon's error handling (Daemon.fatalError) kicks in. It sets the Daemon as shutdown and tries to terminate executor via executor.awaitTermination. This errors out with IllegalMonitorStateException (a RuntimeException)
3. Daemon.handler.onError is therefore bypassed due to RuntimeException. This handler recreates a new Daemon.
4. Subsequent calls to Daemon.add are errored out because daemon is marked shutdown.

Note:

• We're using version 0.12.9 of KPL.

(cc @jcchen2)