openvidu
openvidu copied to clipboard
OpenVidu
openvidu pro losing recordings
Some recordings keep with 0mbs, also this session haven't history events in openvidu inspector..
I am looking the logs in kibana and show it ->
rtpsource rtpsource.c:1147 update_receiver_stats() duplicate or reordered packet (seqnr 26563, expected 26565) 01:33:38.166 rtpjitterbuffer rtpjitterbuffer.c:735 rtp_jitter_buffer_insert() rtp delta too big, reset skew 01:33:38.214 {"type": "server", "timestamp": "2021-01-09T04:33:38,214Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2020.12.08] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [197]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.214 {"type": "server", "timestamp": "2021-01-09T04:33:38,214Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2020.12.30] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [549]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.214 {"type": "server", "timestamp": "2021-01-09T04:33:38,214Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2020.12.08] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [197]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.214 {"type": "server", "timestamp": "2021-01-09T04:33:38,214Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2020.12.30] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [549]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.215 {"type": "server", "timestamp": "2021-01-09T04:33:38,215Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2020.12.11] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [339]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.215 {"type": "server", "timestamp": "2021-01-09T04:33:38,215Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2020.12.23] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [1053]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.215 {"type": "server", "timestamp": "2021-01-09T04:33:38,215Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2020.12.24] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [981]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.215 {"type": "server", "timestamp": "2021-01-09T04:33:38,214Z", "level": "ERROR", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "retry execution of step [check-rollover-ready] for index [metricbeat-7.8.0-2020.12.08] failed", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" , 01:33:38.215 "stacktrace": ["java.lang.IllegalArgumentException: step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] for index [metricbeat-7.8.0-2020.12.08] with policy [openvidu_cleanup_policy] does not exist", 01:33:38.215 "at org.elasticsearch.xpack.ilm.IndexLifecycleTransition.validateTransition(IndexLifecycleTransition.java:84) ~[?:?]", 01:33:38.215 "at org.elasticsearch.xpack.ilm.IndexLifecycleTransition.moveClusterStateToPreviouslyFailedStep(IndexLifecycleTransition.java:186) ~[?:?]", 01:33:38.215 "at org.elasticsearch.xpack.ilm.IndexLifecycleRunner$2.execute(IndexLifecycleRunner.java:230) ~[?:?]", 01:33:38.215 "at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.215 "at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:702) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.215 "at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:324) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.215 "at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:219) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.215 "at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.215 "at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.215 "at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.215 "at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.215 "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:636) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.215 "at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.215 "at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.215 "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]", 01:33:38.215 "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]", 01:33:38.215 "at java.lang.Thread.run(Thread.java:832) [?:?]"] } 01:33:38.215 {"type": "server", "timestamp": "2021-01-09T04:33:38,215Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2020.12.11] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [339]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.215 {"type": "server", "timestamp": "2021-01-09T04:33:38,215Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2020.12.23] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [1053]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.215 {"type": "server", "timestamp": "2021-01-09T04:33:38,215Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2020.12.24] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [981]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.215 {"type": "server", "timestamp": "2021-01-09T04:33:38,214Z", "level": "ERROR", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "retry execution of step [check-rollover-ready] for index [metricbeat-7.8.0-2020.12.08] failed", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" , 01:33:38.215 "stacktrace": ["java.lang.IllegalArgumentException: step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] for index [metricbeat-7.8.0-2020.12.08] with policy [openvidu_cleanup_policy] does not exist", 01:33:38.215 "at org.elasticsearch.xpack.ilm.IndexLifecycleTransition.validateTransition(IndexLifecycleTransition.java:84) ~[?:?]", 01:33:38.215 "at org.elasticsearch.xpack.ilm.IndexLifecycleTransition.moveClusterStateToPreviouslyFailedStep(IndexLifecycleTransition.java:186) ~[?:?]", 01:33:38.215 "at org.elasticsearch.xpack.ilm.IndexLifecycleRunner$2.execute(IndexLifecycleRunner.java:230) ~[?:?]", 01:33:38.215 "at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.215 "at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:702) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.215 "at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:324) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.215 "at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:219) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.215 "at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.215 "at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.215 "at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.215 "at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.215 "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:636) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.215 "at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.215 "at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.215 "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]", 01:33:38.215 "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]", 01:33:38.215 "at java.lang.Thread.run(Thread.java:832) [?:?]"] } 01:33:38.216 {"type": "server", "timestamp": "2021-01-09T04:33:38,216Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2020.12.21] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [1197]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.216 {"type": "server", "timestamp": "2021-01-09T04:33:38,215Z", "level": "ERROR", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "retry execution of step [check-rollover-ready] for index [metricbeat-7.8.0-2020.12.30] failed", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" , 01:33:38.216 "stacktrace": ["java.lang.IllegalArgumentException: step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] for index [metricbeat-7.8.0-2020.12.30] with policy [openvidu_cleanup_policy] does not exist", 01:33:38.216 "at org.elasticsearch.xpack.ilm.IndexLifecycleTransition.validateTransition(IndexLifecycleTransition.java:84) ~[?:?]", 01:33:38.216 "at org.elasticsearch.xpack.ilm.IndexLifecycleTransition.moveClusterStateToPreviouslyFailedStep(IndexLifecycleTransition.java:186) ~[?:?]", 01:33:38.216 "at org.elasticsearch.xpack.ilm.IndexLifecycleRunner$2.execute(IndexLifecycleRunner.java:230) ~[?:?]", 01:33:38.216 "at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.216 "at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:702) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.216 "at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:324) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.216 "at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:219) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.216 "at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.216 "at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.216 "at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.216 "at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.216 "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:636) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.216 "at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.216 "at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.216 "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]", 01:33:38.216 "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]", 01:33:38.216 "at java.lang.Thread.run(Thread.java:832) [?:?]"] } 01:33:38.216 {"type": "server", "timestamp": "2021-01-09T04:33:38,216Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2020.12.27] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [765]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.216 {"type": "server", "timestamp": "2021-01-09T04:33:38,216Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2020.12.19] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [1341]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.216 {"type": "server", "timestamp": "2021-01-09T04:33:38,216Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2020.12.21] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [1197]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.216 {"type": "server", "timestamp": "2021-01-09T04:33:38,215Z", "level": "ERROR", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "retry execution of step [check-rollover-ready] for index [metricbeat-7.8.0-2020.12.30] failed", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" , 01:33:38.216 "stacktrace": ["java.lang.IllegalArgumentException: step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] for index [metricbeat-7.8.0-2020.12.30] with policy [openvidu_cleanup_policy] does not exist", 01:33:38.216 "at org.elasticsearch.xpack.ilm.IndexLifecycleTransition.validateTransition(IndexLifecycleTransition.java:84) ~[?:?]", 01:33:38.216 "at org.elasticsearch.xpack.ilm.IndexLifecycleTransition.moveClusterStateToPreviouslyFailedStep(IndexLifecycleTransition.java:186) ~[?:?]", 01:33:38.216 "at org.elasticsearch.xpack.ilm.IndexLifecycleRunner$2.execute(IndexLifecycleRunner.java:230) ~[?:?]", 01:33:38.216 "at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.216 "at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:702) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.216 "at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:324) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.216 "at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:219) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.216 "at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.216 "at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.216 "at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.216 "at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.216 "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:636) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.216 "at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.216 "at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.216 "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]", 01:33:38.216 "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]", 01:33:38.216 "at java.lang.Thread.run(Thread.java:832) [?:?]"] } 01:33:38.216 {"type": "server", "timestamp": "2021-01-09T04:33:38,216Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2020.12.27] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [765]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.216 {"type": "server", "timestamp": "2021-01-09T04:33:38,216Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2020.12.19] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [1341]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.217 {"type": "server", "timestamp": "2021-01-09T04:33:38,216Z", "level": "ERROR", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "retry execution of step [check-rollover-ready] for index [metricbeat-7.8.0-2020.12.11] failed", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" , 01:33:38.217 "stacktrace": ["java.lang.IllegalArgumentException: step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] for index [metricbeat-7.8.0-2020.12.11] with policy [openvidu_cleanup_policy] does not exist", 01:33:38.217 "at org.elasticsearch.xpack.ilm.IndexLifecycleTransition.validateTransition(IndexLifecycleTransition.java:84) ~[?:?]", 01:33:38.217 "at org.elasticsearch.xpack.ilm.IndexLifecycleTransition.moveClusterStateToPreviouslyFailedStep(IndexLifecycleTransition.java:186) ~[?:?]", 01:33:38.217 "at org.elasticsearch.xpack.ilm.IndexLifecycleRunner$2.execute(IndexLifecycleRunner.java:230) ~[?:?]", 01:33:38.217 "at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:702) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:324) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:219) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:636) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]", 01:33:38.217 "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]", 01:33:38.217 "at java.lang.Thread.run(Thread.java:832) [?:?]"] } 01:33:38.217 {"type": "server", "timestamp": "2021-01-09T04:33:38,216Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2020.12.17] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [1485]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.217 {"type": "server", "timestamp": "2021-01-09T04:33:38,217Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2020.12.10] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [53]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.217 {"type": "server", "timestamp": "2021-01-09T04:33:38,217Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2020.12.28] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [693]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.217 {"type": "server", "timestamp": "2021-01-09T04:33:38,217Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2021.01.01] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [405]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.217 {"type": "server", "timestamp": "2021-01-09T04:33:38,217Z", "level": "ERROR", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "retry execution of step [check-rollover-ready] for index [metricbeat-7.8.0-2020.12.23] failed", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" , 01:33:38.217 "stacktrace": ["java.lang.IllegalArgumentException: step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] for index [metricbeat-7.8.0-2020.12.23] with policy [openvidu_cleanup_policy] does not exist", 01:33:38.217 "at org.elasticsearch.xpack.ilm.IndexLifecycleTransition.validateTransition(IndexLifecycleTransition.java:84) ~[?:?]", 01:33:38.217 "at org.elasticsearch.xpack.ilm.IndexLifecycleTransition.moveClusterStateToPreviouslyFailedStep(IndexLifecycleTransition.java:186) ~[?:?]", 01:33:38.217 "at org.elasticsearch.xpack.ilm.IndexLifecycleRunner$2.execute(IndexLifecycleRunner.java:230) ~[?:?]", 01:33:38.217 "at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:702) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:324) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:219) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:636) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]", 01:33:38.217 "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]", 01:33:38.217 "at java.lang.Thread.run(Thread.java:832) [?:?]"] } 01:33:38.217 {"type": "server", "timestamp": "2021-01-09T04:33:38,217Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2020.12.03] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [557]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.217 {"type": "server", "timestamp": "2021-01-09T04:33:38,216Z", "level": "ERROR", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "retry execution of step [check-rollover-ready] for index [metricbeat-7.8.0-2020.12.11] failed", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" , 01:33:38.217 "stacktrace": ["java.lang.IllegalArgumentException: step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] for index [metricbeat-7.8.0-2020.12.11] with policy [openvidu_cleanup_policy] does not exist", 01:33:38.217 "at org.elasticsearch.xpack.ilm.IndexLifecycleTransition.validateTransition(IndexLifecycleTransition.java:84) ~[?:?]", 01:33:38.217 "at org.elasticsearch.xpack.ilm.IndexLifecycleTransition.moveClusterStateToPreviouslyFailedStep(IndexLifecycleTransition.java:186) ~[?:?]", 01:33:38.217 "at org.elasticsearch.xpack.ilm.IndexLifecycleRunner$2.execute(IndexLifecycleRunner.java:230) ~[?:?]", 01:33:38.217 "at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:702) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:324) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:219) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:636) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]", 01:33:38.217 "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]", 01:33:38.217 "at java.lang.Thread.run(Thread.java:832) [?:?]"] } 01:33:38.217 {"type": "server", "timestamp": "2021-01-09T04:33:38,216Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2020.12.17] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [1485]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.217 {"type": "server", "timestamp": "2021-01-09T04:33:38,217Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2020.12.10] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [53]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.217 {"type": "server", "timestamp": "2021-01-09T04:33:38,217Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2020.12.28] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [693]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.217 {"type": "server", "timestamp": "2021-01-09T04:33:38,217Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2021.01.01] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [405]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.217 {"type": "server", "timestamp": "2021-01-09T04:33:38,217Z", "level": "ERROR", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "retry execution of step [check-rollover-ready] for index [metricbeat-7.8.0-2020.12.23] failed", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" , 01:33:38.217 "stacktrace": ["java.lang.IllegalArgumentException: step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] for index [metricbeat-7.8.0-2020.12.23] with policy [openvidu_cleanup_policy] does not exist", 01:33:38.217 "at org.elasticsearch.xpack.ilm.IndexLifecycleTransition.validateTransition(IndexLifecycleTransition.java:84) ~[?:?]", 01:33:38.217 "at org.elasticsearch.xpack.ilm.IndexLifecycleTransition.moveClusterStateToPreviouslyFailedStep(IndexLifecycleTransition.java:186) ~[?:?]", 01:33:38.217 "at org.elasticsearch.xpack.ilm.IndexLifecycleRunner$2.execute(IndexLifecycleRunner.java:230) ~[?:?]", 01:33:38.217 "at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:702) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:324) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:219) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:636) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.217 "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]", 01:33:38.217 "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]", 01:33:38.217 "at java.lang.Thread.run(Thread.java:832) [?:?]"] } 01:33:38.217 {"type": "server", "timestamp": "2021-01-09T04:33:38,217Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2020.12.03] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [557]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.218 {"type": "server", "timestamp": "2021-01-09T04:33:38,218Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2020.12.18] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [1413]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.218 {"type": "server", "timestamp": "2021-01-09T04:33:38,217Z", "level": "ERROR", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "retry execution of step [check-rollover-ready] for index [metricbeat-7.8.0-2020.12.24] failed", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" , 01:33:38.218 "stacktrace": ["java.lang.IllegalArgumentException: step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] for index [metricbeat-7.8.0-2020.12.24] with policy [openvidu_cleanup_policy] does not exist", 01:33:38.218 "at org.elasticsearch.xpack.ilm.IndexLifecycleTransition.validateTransition(IndexLifecycleTransition.java:84) ~[?:?]", 01:33:38.218 "at org.elasticsearch.xpack.ilm.IndexLifecycleTransition.moveClusterStateToPreviouslyFailedStep(IndexLifecycleTransition.java:186) ~[?:?]", 01:33:38.218 "at org.elasticsearch.xpack.ilm.IndexLifecycleRunner$2.execute(IndexLifecycleRunner.java:230) ~[?:?]", 01:33:38.218 "at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.218 "at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:702) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.218 "at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:324) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.218 "at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:219) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.218 "at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.218 "at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.218 "at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.218 "at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.218 "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:636) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.218 "at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.218 "at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.218 "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]", 01:33:38.218 "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]", 01:33:38.218 "at java.lang.Thread.run(Thread.java:832) [?:?]"] } 01:33:38.218 {"type": "server", "timestamp": "2021-01-09T04:33:38,218Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2020.12.12] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [267]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.218 {"type": "server", "timestamp": "2021-01-09T04:33:38,218Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2020.12.15] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [51]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.218 {"type": "server", "timestamp": "2021-01-09T04:33:38,218Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2020.12.18] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [1413]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.218 {"type": "server", "timestamp": "2021-01-09T04:33:38,217Z", "level": "ERROR", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "retry execution of step [check-rollover-ready] for index [metricbeat-7.8.0-2020.12.24] failed", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" , 01:33:38.218 "stacktrace": ["java.lang.IllegalArgumentException: step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] for index [metricbeat-7.8.0-2020.12.24] with policy [openvidu_cleanup_policy] does not exist", 01:33:38.218 "at org.elasticsearch.xpack.ilm.IndexLifecycleTransition.validateTransition(IndexLifecycleTransition.java:84) ~[?:?]", 01:33:38.218 "at org.elasticsearch.xpack.ilm.IndexLifecycleTransition.moveClusterStateToPreviouslyFailedStep(IndexLifecycleTransition.java:186) ~[?:?]", 01:33:38.218 "at org.elasticsearch.xpack.ilm.IndexLifecycleRunner$2.execute(IndexLifecycleRunner.java:230) ~[?:?]", 01:33:38.218 "at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.218 "at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:702) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.218 "at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:324) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.218 "at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:219) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.218 "at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.218 "at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.218 "at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.218 "at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.218 "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:636) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.218 "at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.218 "at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.218 "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]", 01:33:38.218 "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]", 01:33:38.218 "at java.lang.Thread.run(Thread.java:832) [?:?]"] } 01:33:38.218 {"type": "server", "timestamp": "2021-01-09T04:33:38,218Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2020.12.12] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [267]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.218 {"type": "server", "timestamp": "2021-01-09T04:33:38,218Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2020.12.15] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [51]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.219 {"type": "server", "timestamp": "2021-01-09T04:33:38,218Z", "level": "ERROR", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "retry execution of step [check-rollover-ready] for index [metricbeat-7.8.0-2020.12.21] failed", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" , 01:33:38.219 "stacktrace": ["java.lang.IllegalArgumentException: step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] for index [metricbeat-7.8.0-2020.12.21] with policy [openvidu_cleanup_policy] does not exist", 01:33:38.219 "at org.elasticsearch.xpack.ilm.IndexLifecycleTransition.validateTransition(IndexLifecycleTransition.java:84) ~[?:?]", 01:33:38.219 "at org.elasticsearch.xpack.ilm.IndexLifecycleTransition.moveClusterStateToPreviouslyFailedStep(IndexLifecycleTransition.java:186) ~[?:?]", 01:33:38.219 "at org.elasticsearch.xpack.ilm.IndexLifecycleRunner$2.execute(IndexLifecycleRunner.java:230) ~[?:?]", 01:33:38.219 "at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:702) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:324) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:219) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:636) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]", 01:33:38.219 "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]", 01:33:38.219 "at java.lang.Thread.run(Thread.java:832) [?:?]"] } 01:33:38.219 {"type": "server", "timestamp": "2021-01-09T04:33:38,218Z", "level": "ERROR", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [metricbeat] for index [metricbeat-7.8.0-2021.01.08] failed on step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}]. Moving to ERROR step", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" , 01:33:38.219 "stacktrace": ["java.lang.IllegalArgumentException: index.lifecycle.rollover_alias [metricbeat-7.8.0] does not point to index [metricbeat-7.8.0-2021.01.08]", 01:33:38.219 "at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:104) [x-pack-core-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:173) [x-pack-ilm-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:329) [x-pack-ilm-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:267) [x-pack-ilm-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:183) [x-pack-core-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:211) [x-pack-core-7.8.0.jar:7.8.0]", 01:33:38.219 "at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]", 01:33:38.219 "at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]", 01:33:38.219 "at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]", 01:33:38.219 "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]", 01:33:38.219 "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]", 01:33:38.219 "at java.lang.Thread.run(Thread.java:832) [?:?]"] } 01:33:38.219 {"type": "server", "timestamp": "2021-01-09T04:33:38,218Z", "level": "ERROR", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "retry execution of step [check-rollover-ready] for index [metricbeat-7.8.0-2020.12.27] failed", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" , 01:33:38.219 "stacktrace": ["java.lang.IllegalArgumentException: step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] for index [metricbeat-7.8.0-2020.12.27] with policy [openvidu_cleanup_policy] does not exist", 01:33:38.219 "at org.elasticsearch.xpack.ilm.IndexLifecycleTransition.validateTransition(IndexLifecycleTransition.java:84) ~[?:?]", 01:33:38.219 "at org.elasticsearch.xpack.ilm.IndexLifecycleTransition.moveClusterStateToPreviouslyFailedStep(IndexLifecycleTransition.java:186) ~[?:?]", 01:33:38.219 "at org.elasticsearch.xpack.ilm.IndexLifecycleRunner$2.execute(IndexLifecycleRunner.java:230) ~[?:?]", 01:33:38.219 "at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:702) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:324) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:219) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:636) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]", 01:33:38.219 "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]", 01:33:38.219 "at java.lang.Thread.run(Thread.java:832) [?:?]"] } 01:33:38.219 {"type": "server", "timestamp": "2021-01-09T04:33:38,219Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2020.12.09] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [125]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.219 {"type": "server", "timestamp": "2021-01-09T04:33:38,219Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2020.12.26] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [837]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.219 {"type": "server", "timestamp": "2021-01-09T04:33:38,218Z", "level": "ERROR", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "retry execution of step [check-rollover-ready] for index [metricbeat-7.8.0-2020.12.21] failed", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" , 01:33:38.219 "stacktrace": ["java.lang.IllegalArgumentException: step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] for index [metricbeat-7.8.0-2020.12.21] with policy [openvidu_cleanup_policy] does not exist", 01:33:38.219 "at org.elasticsearch.xpack.ilm.IndexLifecycleTransition.validateTransition(IndexLifecycleTransition.java:84) ~[?:?]", 01:33:38.219 "at org.elasticsearch.xpack.ilm.IndexLifecycleTransition.moveClusterStateToPreviouslyFailedStep(IndexLifecycleTransition.java:186) ~[?:?]", 01:33:38.219 "at org.elasticsearch.xpack.ilm.IndexLifecycleRunner$2.execute(IndexLifecycleRunner.java:230) ~[?:?]", 01:33:38.219 "at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:702) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:324) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:219) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:636) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]", 01:33:38.219 "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]", 01:33:38.219 "at java.lang.Thread.run(Thread.java:832) [?:?]"] } 01:33:38.219 {"type": "server", "timestamp": "2021-01-09T04:33:38,218Z", "level": "ERROR", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [metricbeat] for index [metricbeat-7.8.0-2021.01.08] failed on step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}]. Moving to ERROR step", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" , 01:33:38.219 "stacktrace": ["java.lang.IllegalArgumentException: index.lifecycle.rollover_alias [metricbeat-7.8.0] does not point to index [metricbeat-7.8.0-2021.01.08]", 01:33:38.219 "at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:104) [x-pack-core-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:173) [x-pack-ilm-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:329) [x-pack-ilm-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:267) [x-pack-ilm-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:183) [x-pack-core-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:211) [x-pack-core-7.8.0.jar:7.8.0]", 01:33:38.219 "at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]", 01:33:38.219 "at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]", 01:33:38.219 "at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]", 01:33:38.219 "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]", 01:33:38.219 "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]", 01:33:38.219 "at java.lang.Thread.run(Thread.java:832) [?:?]"] } 01:33:38.219 {"type": "server", "timestamp": "2021-01-09T04:33:38,218Z", "level": "ERROR", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "retry execution of step [check-rollover-ready] for index [metricbeat-7.8.0-2020.12.27] failed", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" , 01:33:38.219 "stacktrace": ["java.lang.IllegalArgumentException: step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] for index [metricbeat-7.8.0-2020.12.27] with policy [openvidu_cleanup_policy] does not exist", 01:33:38.219 "at org.elasticsearch.xpack.ilm.IndexLifecycleTransition.validateTransition(IndexLifecycleTransition.java:84) ~[?:?]", 01:33:38.219 "at org.elasticsearch.xpack.ilm.IndexLifecycleTransition.moveClusterStateToPreviouslyFailedStep(IndexLifecycleTransition.java:186) ~[?:?]", 01:33:38.219 "at org.elasticsearch.xpack.ilm.IndexLifecycleRunner$2.execute(IndexLifecycleRunner.java:230) ~[?:?]", 01:33:38.219 "at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:702) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:324) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:219) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:636) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.219 "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]", 01:33:38.219 "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]", 01:33:38.219 "at java.lang.Thread.run(Thread.java:832) [?:?]"] } 01:33:38.219 {"type": "server", "timestamp": "2021-01-09T04:33:38,219Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2020.12.09] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [125]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.219 {"type": "server", "timestamp": "2021-01-09T04:33:38,219Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2020.12.26] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [837]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.220 {"type": "server", "timestamp": "2021-01-09T04:33:38,219Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2020.12.06] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [341]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.220 {"type": "server", "timestamp": "2021-01-09T04:33:38,220Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2021.01.06] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [45]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.220 {"type": "server", "timestamp": "2021-01-09T04:33:38,219Z", "level": "ERROR", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "retry execution of step [check-rollover-ready] for index [metricbeat-7.8.0-2020.12.19] failed", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" , 01:33:38.220 "stacktrace": ["java.lang.IllegalArgumentException: step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] for index [metricbeat-7.8.0-2020.12.19] with policy [openvidu_cleanup_policy] does not exist", 01:33:38.220 "at org.elasticsearch.xpack.ilm.IndexLifecycleTransition.validateTransition(IndexLifecycleTransition.java:84) ~[?:?]", 01:33:38.220 "at org.elasticsearch.xpack.ilm.IndexLifecycleTransition.moveClusterStateToPreviouslyFailedStep(IndexLifecycleTransition.java:186) ~[?:?]", 01:33:38.220 "at org.elasticsearch.xpack.ilm.IndexLifecycleRunner$2.execute(IndexLifecycleRunner.java:230) ~[?:?]", 01:33:38.220 "at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.220 "at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:702) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.220 "at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:324) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.220 "at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:219) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.220 "at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.220 "at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.220 "at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.220 "at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.220 "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:636) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.220 "at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.220 "at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.220 "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]", 01:33:38.220 "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]", 01:33:38.220 "at java.lang.Thread.run(Thread.java:832) [?:?]"] } 01:33:38.220 {"type": "server", "timestamp": "2021-01-09T04:33:38,220Z", "level": "ERROR", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [metricbeat] for index [metricbeat-7.8.0-2021.01.07] failed on step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}]. Moving to ERROR step", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" , 01:33:38.220 "stacktrace": ["java.lang.IllegalArgumentException: index.lifecycle.rollover_alias [metricbeat-7.8.0] does not point to index [metricbeat-7.8.0-2021.01.07]", 01:33:38.220 "at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:104) [x-pack-core-7.8.0.jar:7.8.0]", 01:33:38.220 "at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:173) [x-pack-ilm-7.8.0.jar:7.8.0]", 01:33:38.220 "at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:329) [x-pack-ilm-7.8.0.jar:7.8.0]", 01:33:38.220 "at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:267) [x-pack-ilm-7.8.0.jar:7.8.0]", 01:33:38.220 "at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:183) [x-pack-core-7.8.0.jar:7.8.0]", 01:33:38.220 "at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:211) [x-pack-core-7.8.0.jar:7.8.0]", 01:33:38.220 "at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]", 01:33:38.220 "at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]", 01:33:38.220 "at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]", 01:33:38.220 "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]", 01:33:38.220 "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]", 01:33:38.220 "at java.lang.Thread.run(Thread.java:832) [?:?]"] } 01:33:38.220 {"type": "server", "timestamp": "2021-01-09T04:33:38,220Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2020.12.22] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [1125]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.220 {"type": "server", "timestamp": "2021-01-09T04:33:38,220Z", "level": "ERROR", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "retry execution of step [check-rollover-ready] for index [metricbeat-7.8.0-2020.12.17] failed", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" , 01:33:38.220 "stacktrace": ["java.lang.IllegalArgumentException: step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] for index [metricbeat-7.8.0-2020.12.17] with policy [openvidu_cleanup_policy] does not exist", 01:33:38.220 "at org.elasticsearch.xpack.ilm.IndexLifecycleTransition.validateTransition(IndexLifecycleTransition.java:84) ~[?:?]", 01:33:38.220 "at org.elasticsearch.xpack.ilm.IndexLifecycleTransition.moveClusterStateToPreviouslyFailedStep(IndexLifecycleTransition.java:186) ~[?:?]", 01:33:38.220 "at org.elasticsearch.xpack.ilm.IndexLifecycleRunner$2.execute(IndexLifecycleRunner.java:230) ~[?:?]", 01:33:38.220 "at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.220 "at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:702) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.220 {"type": "server", "timestamp": "2021-01-09T04:33:38,219Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2020.12.06] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [341]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.220 {"type": "server", "timestamp": "2021-01-09T04:33:38,220Z", "level": "INFO", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "policy [openvidu_cleanup_policy] for index [metricbeat-7.8.0-2021.01.06] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [45]", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" } 01:33:38.220 {"type": "server", "timestamp": "2021-01-09T04:33:38,219Z", "level": "ERROR", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "docker-cluster", "node.name": "25d3f442c9ad", "message": "retry execution of step [check-rollover-ready] for index [metricbeat-7.8.0-2020.12.19] failed", "cluster.uuid": "xK3wJJOJS1W1Ph_goViE2Q", "node.id": "Vb5Serx_R56wnwa9Qx2dWw" , 01:33:38.220 "stacktrace": ["java.lang.IllegalArgumentException: step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] for index [metricbeat-7.8.0-2020.12.19] with policy [openvidu_cleanup_policy] does not exist", 01:33:38.220 "at org.elasticsearch.xpack.ilm.IndexLifecycleTransition.validateTransition(IndexLifecycleTransition.java:84) ~[?:?]", 01:33:38.220 "at org.elasticsearch.xpack.ilm.IndexLifecycleTransition.moveClusterStateToPreviouslyFailedStep(IndexLifecycleTransition.java:186) ~[?:?]", 01:33:38.220 "at org.elasticsearch.xpack.ilm.IndexLifecycleRunner$2.execute(IndexLifecycleRunner.java:230) ~[?:?]", 01:33:38.220 "at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.220 "at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:702) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.220 "at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:324) ~[elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.220 "at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:219) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.220 "at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.220 "at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.220 "at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.220 "at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.220 "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:636) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.220 "at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.220 "at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.8.0.jar:7.8.0]", 01:33:38.220 "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]", 01:33:38.220 "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]", 01:33:38.220 "at java.lang.Thread.run(Thread.java:832) [?:?]"] } 01:33:38.220
It seems ElasticSearch has had an issue.
Can you found KMS log during the recording? KMS is responsible of recording and maybe there is something in the log helping us to catch the error
