ILM not working as expected

[JZ-SmartThings]

trafficstars

Had the ES service go down... finding the below errors and state of affairs. Seems to relate to ILM failing.

[2020-06-08T07:09:17,718][ERROR][o.e.x.i.IndexLifecycleRunner] [jznode] policy [ilm-history-ilm-policy] for index [ilm-history-2-000001] failed on step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}]. Moving to ERROR step org.elasticsearch.discovery.MasterNotDiscoveredException: null at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) [elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.cluster.ClusterStateObserver.waitForNextChange(ClusterStateObserver.java:149) [elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.cluster.ClusterStateObserver.waitForNextChange(ClusterStateObserver.java:120) [elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.retry(TransportMasterNodeAction.java:204) [elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:175) [elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.start(TransportMasterNodeAction.java:133) [elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:110) [elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:59) [elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:153) [elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:123) [x-pack-security-7.7.0.jar:7.7.0] at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:151) [elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:129) [elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:64) [elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) [elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) [elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:399) [elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.xpack.core.ClientHelper.executeAsyncWithOrigin(ClientHelper.java:92) [x-pack-core-7.7.0.jar:7.7.0] at org.elasticsearch.xpack.core.ClientHelper.executeWithHeadersAsync(ClientHelper.java:155) [x-pack-core-7.7.0.jar:7.7.0] at org.elasticsearch.xpack.ilm.LifecyclePolicySecurityClient.doExecute(LifecyclePolicySecurityClient.java:51) [x-pack-ilm-7.7.0.jar:7.7.0] at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:399) [elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1234) [elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.rolloverIndex(AbstractClient.java:1736) [elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:127) [x-pack-core-7.7.0.jar:7.7.0] at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:173) [x-pack-ilm-7.7.0.jar:7.7.0] at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:329) [x-pack-ilm-7.7.0.jar:7.7.0] at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:267) [x-pack-ilm-7.7.0.jar:7.7.0] at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:183) [x-pack-core-7.7.0.jar:7.7.0] at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:211) [x-pack-core-7.7.0.jar:7.7.0] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?] at java.lang.Thread.run(Thread.java:832) [?:?]

[legrego]

I started researching this, but don't have a definitive root cause at this point

[JZ-SmartThings]

thanks for the update @legrego!

[legrego]

I'm going to close this as "not planned" since I've been unable to reproduce this, and it does not appear to be a widespread issue. I'm happy to investigate further if this continues to be a problem.