Seeing interrupted exception when closing PathChildrenCache

[keith-turner]

After patch for #206 seeing alot of the following exceptions. Looking into this, @cjnolet may have had the issue fixed in CURATOR-121. Going to curator 2.7 may make these exceptions go away.

2015-02-07 11:20:44,152 [server.AbstractNonblockingServer$FrameBuffer] ERROR: Unexpected throwable while invoking!
java.lang.IllegalStateException
    at io.fluo.core.oracle.OracleServer._getTimestamps(OracleServer.java:158)
    at io.fluo.core.oracle.OracleServer.getTimestamps(OracleServer.java:141)
    at io.fluo.core.thrift.OracleService$Processor$getTimestamps.getResult(OracleService.java:224)
    at io.fluo.core.thrift.OracleService$Processor$getTimestamps.getResult(OracleService.java:209)
    at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
    at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:516)
    at org.apache.thrift.server.Invocation.run(Invocation.java:18)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
2015-02-07 11:20:45,887 [cache.PathChildrenCache] ERROR: 
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /oracle/server/_c_31687f18-40ca-4ec2-881d-7a08a67028c8-lock-0000000000
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
    at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:302)
    at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:291)
    at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
    at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:287)
    at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
    at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
    at org.apache.curator.framework.recipes.leader.LeaderSelector.participantForPath(LeaderSelector.java:374)
    at org.apache.curator.framework.recipes.leader.LeaderSelector.getLeader(LeaderSelector.java:345)
    at org.apache.curator.framework.recipes.leader.LeaderSelector.getLeader(LeaderSelector.java:338)
    at io.fluo.core.oracle.OracleServer.childEvent(OracleServer.java:330)
    at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:509)
    at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:503)
    at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92)
    at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:262)
    at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83)
    at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:500)
    at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35)
    at org.apache.curator.framework.recipes.cache.PathChildrenCache$10.run(PathChildrenCache.java:762)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

[mikewalch]

I upgraded Curator to 2.7.1. While CURATOR-21 removed the logging of this exception at one spot in the class, there are still more places where this exception can be unessarily logged. As this issue is only seen during Fluo ITs, I submitted a PR to increase the log level for this class to FATAL for ITs. However, this issue can be kept open until a long term solution is found.

[mikewalch]

With the log level increase commit, this issue will be removed from beta milestone but will not be closed until a long-term fix is made to curator.