Cluster stuck in voting loop?

Open jkassis opened this issue 4 years ago • 20 comments

This could be a tuning problem, but it seems that my raft cluster has been stuck in a voting loop for quite some time. This surfaced in production when a SyncRead timed out after 10secs.

Dragonboat version

github.com/lni/dragonboat/v3 v3.2.8

Expected behavior

Voting protocol elects leader

Actual behavior

blah. blah. blah.

time="2021-07-26T15:22:52Z" level=info msg="Dockie: Sync" file=" Jul 26 15:22:52.310577199 m/jkassis/[email protected]/logger.go: 192 " time="2021-07-26T15:26:28Z" level=info msg="Dockie: Sync" file=" Jul 26 15:26:28.392497518 m/jkassis/[email protected]/logger.go: 192 " time="2021-07-26T15:30:07Z" level=info msg="Dockie: Sync" file=" Jul 26 15:30:07.871855858 m/jkassis/[email protected]/logger.go: 192 " time="2021-07-26T15:33:45Z" level=info msg="Dockie: Sync" file=" Jul 26 15:33:45.429222303 m/jkassis/[email protected]/logger.go: 192 " time="2021-07-26T15:35:18Z" level=warning msg="Dockie: IndexDocsOp: put: per" file=" Jul 26 15:35:18.047395994 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:35:45Z" level=warning msg="[f:8199,l:10528,t:2044,c:10528,a:10528] [00010:51496] t2044 received RequestVote with higher term (2045) from n01307" file=" Jul 26 15:35:45.324971684 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:35:45Z" level=warning msg="[f:8199,l:10528,t:2044,c:10528,a:10528] [00010:51496] t2044 become followerKE after receiving higher term from n01307" file=" Jul 26 15:35:45.325040065 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:35:45Z" level=warning msg="[f:8199,l:10528,t:2044,c:10528,a:10528] [00010:51496] t2045 cast vote from n01307 index 10528 term 2045, log term: 2044" file=" Jul 26 15:35:45.325110456 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:35:51Z" level=warning msg="[f:8199,l:10529,t:2045,c:10528,a:10528] [00010:51496] t2046 became candidate" file=" Jul 26 15:35:51.691058985 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:35:51Z" level=warning msg="[f:8199,l:10529,t:2045,c:10528,a:10528] [00010:51496] t2046 received RequestVoteResp from n51496" file=" Jul 26 15:35:51.691128566 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:35:51Z" level=warning msg="[f:8199,l:10529,t:2045,c:10528,a:10528] [00010:51496] t2046 sent RequestVote to n01307" file=" Jul 26 15:35:51.691170096 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:35:51Z" level=warning msg="[f:8199,l:10529,t:2045,c:10528,a:10528] [00010:51496] t2046 sent RequestVote to n60202" file=" Jul 26 15:35:51.691203796 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:35:54Z" level=warning msg="[f:8199,l:10529,t:2045,c:10528,a:10528] [00010:51496] t2047 became candidate" file=" Jul 26 15:35:54.639910468 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:35:54Z" level=warning msg="[f:8199,l:10529,t:2045,c:10528,a:10528] [00010:51496] t2047 received RequestVoteResp from n51496" file=" Jul 26 15:35:54.639983888 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:35:54Z" level=warning msg="[f:8199,l:10529,t:2045,c:10528,a:10528] [00010:51496] t2047 sent RequestVote to n60202" file=" Jul 26 15:35:54.640027919 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:35:54Z" level=warning msg="[f:8199,l:10529,t:2045,c:10528,a:10528] [00010:51496] t2047 sent RequestVote to n01307" file=" Jul 26 15:35:54.640062639 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:35:57Z" level=warning msg="[f:8199,l:10529,t:2045,c:10528,a:10528] [00010:51496] t2048 became candidate" file=" Jul 26 15:35:57.949115475 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:35:57Z" level=warning msg="[f:8199,l:10529,t:2045,c:10528,a:10528] [00010:51496] t2048 received RequestVoteResp from n51496" file=" Jul 26 15:35:57.949189826 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:35:57Z" level=warning msg="[f:8199,l:10529,t:2045,c:10528,a:10528] [00010:51496] t2048 sent RequestVote to n60202" file=" Jul 26 15:35:57.949235366 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:35:57Z" level=warning msg="[f:8199,l:10529,t:2045,c:10528,a:10528] [00010:51496] t2048 sent RequestVote to n01307" file=" Jul 26 15:35:57.949271357 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:02Z" level=warning msg="[f:8199,l:10529,t:2045,c:10528,a:10528] [00010:51496] t2049 became candidate" file=" Jul 26 15:36:02.070015491 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:02Z" level=warning msg="[f:8199,l:10529,t:2045,c:10528,a:10528] [00010:51496] t2049 received RequestVoteResp from n51496" file=" Jul 26 15:36:02.070093432 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:02Z" level=warning msg="[f:8199,l:10529,t:2045,c:10528,a:10528] [00010:51496] t2049 sent RequestVote to n60202" file=" Jul 26 15:36:02.070133302 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:02Z" level=warning msg="[f:8199,l:10529,t:2045,c:10528,a:10528] [00010:51496] t2049 sent RequestVote to n01307" file=" Jul 26 15:36:02.070161103 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:02Z" level=warning msg="[f:8199,l:10529,t:2045,c:10528,a:10528] [00010:51496] t2049 received RequestVoteResp from n60202" file=" Jul 26 15:36:02.206846112 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:02Z" level=warning msg="[f:8199,l:10529,t:2045,c:10528,a:10528] [00010:51496] t2049 received 2 votes and 0 rejections, quorum is 2" file=" Jul 26 15:36:02.206905353 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:20Z" level=warning msg="[f:8199,l:10530,t:2049,c:10530,a:10530] [00010:51496] t2049 received RequestVote with higher term (2050) from n01307" file=" Jul 26 15:36:20.523073191 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:20Z" level=warning msg="[f:8199,l:10530,t:2049,c:10530,a:10530] [00010:51496] t2049 become followerKE after receiving higher term from n01307" file=" Jul 26 15:36:20.523175382 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:20Z" level=warning msg="[f:8199,l:10530,t:2049,c:10530,a:10530] [00010:51496] t2050 cast vote from n01307 index 10530 term 2050, log term: 2049" file=" Jul 26 15:36:20.523235343 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:26Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2051 became candidate" file=" Jul 26 15:36:26.294284195 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:26Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2051 received RequestVoteResp from n51496" file=" Jul 26 15:36:26.294359026 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:26Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2051 sent RequestVote to n01307" file=" Jul 26 15:36:26.294406756 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:26Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2051 sent RequestVote to n60202" file=" Jul 26 15:36:26.294442527 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:30Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2052 became candidate" file=" Jul 26 15:36:30.383911223 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:30Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2052 received RequestVoteResp from n51496" file=" Jul 26 15:36:30.383986454 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:30Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2052 sent RequestVote to n60202" file=" Jul 26 15:36:30.384041674 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:30Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2052 sent RequestVote to n01307" file=" Jul 26 15:36:30.384082815 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:34Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2053 became candidate" file=" Jul 26 15:36:34.773947132 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:34Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2053 received RequestVoteResp from n51496" file=" Jul 26 15:36:34.774038133 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:34Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2053 sent RequestVote to n60202" file=" Jul 26 15:36:34.774088974 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:34Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2053 sent RequestVote to n01307" file=" Jul 26 15:36:34.774130474 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:37Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2054 became candidate" file=" Jul 26 15:36:37.979789289 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:37Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2054 received RequestVoteResp from n51496" file=" Jul 26 15:36:37.979856860 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:37Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2054 sent RequestVote to n60202" file=" Jul 26 15:36:37.979902170 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:37Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2054 sent RequestVote to n01307" file=" Jul 26 15:36:37.979940451 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:41Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2055 became candidate" file=" Jul 26 15:36:41.445812724 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:41Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2055 received RequestVoteResp from n51496" file=" Jul 26 15:36:41.445890425 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:41Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2055 sent RequestVote to n60202" file=" Jul 26 15:36:41.445934525 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:41Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2055 sent RequestVote to n01307" file=" Jul 26 15:36:41.445968505 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:44Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2056 became candidate" file=" Jul 26 15:36:44.186602371 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:44Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2056 received RequestVoteResp from n51496" file=" Jul 26 15:36:44.186686732 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:44Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2056 sent RequestVote to n60202" file=" Jul 26 15:36:44.186729753 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:44Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2056 sent RequestVote to n01307" file=" Jul 26 15:36:44.188110827 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:48Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2057 became candidate" file=" Jul 26 15:36:48.889567170 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:48Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2057 received RequestVoteResp from n51496" file=" Jul 26 15:36:48.889651421 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:48Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2057 sent RequestVote to n60202" file=" Jul 26 15:36:48.889698211 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:48Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2057 sent RequestVote to n01307" file=" Jul 26 15:36:48.889734642 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:52Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2058 became candidate" file=" Jul 26 15:36:52.498907662 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:52Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2058 received RequestVoteResp from n51496" file=" Jul 26 15:36:52.498977393 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:52Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2058 sent RequestVote to n60202" file=" Jul 26 15:36:52.499024114 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:52Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2058 sent RequestVote to n01307" file=" Jul 26 15:36:52.499062934 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:55Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2059 became candidate" file=" Jul 26 15:36:55.306089244 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:55Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2059 received RequestVoteResp from n51496" file=" Jul 26 15:36:55.306159224 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:55Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2059 sent RequestVote to n60202" file=" Jul 26 15:36:55.306206325 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:55Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2059 sent RequestVote to n01307" file=" Jul 26 15:36:55.306243835 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:59Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2060 became candidate" file=" Jul 26 15:36:59.736445491 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:59Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2060 received RequestVoteResp from n51496" file=" Jul 26 15:36:59.736538862 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:59Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2060 sent RequestVote to n60202" file=" Jul 26 15:36:59.736577812 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:36:59Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2060 sent RequestVote to n01307" file=" Jul 26 15:36:59.736605303 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:02Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2060 received RequestVoteResp from n60202" file=" Jul 26 15:37:02.111932873 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:02Z" level=warning msg="[f:8199,l:10531,t:2050,c:10530,a:10530] [00010:51496] t2060 received 2 votes and 0 rejections, quorum is 2" file=" Jul 26 15:37:02.111996953 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:12Z" level=warning msg="[f:8199,l:10532,t:2060,c:10532,a:10532] [00010:51496] t2060 received RequestVote with higher term (2061) from n01307" file=" Jul 26 15:37:12.557302142 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:12Z" level=warning msg="[f:8199,l:10532,t:2060,c:10532,a:10532] [00010:51496] t2060 become followerKE after receiving higher term from n01307" file=" Jul 26 15:37:12.557366323 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:12Z" level=warning msg="[f:8199,l:10532,t:2060,c:10532,a:10532] [00010:51496] t2061 cast vote from n01307 index 10532 term 2061, log term: 2060" file=" Jul 26 15:37:12.557410814 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:24Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2062 became candidate" file=" Jul 26 15:37:24.080590652 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:24Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2062 received RequestVoteResp from n51496" file=" Jul 26 15:37:24.080669062 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:24Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2062 sent RequestVote to n01307" file=" Jul 26 15:37:24.080713093 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:24Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2062 sent RequestVote to n60202" file=" Jul 26 15:37:24.080769114 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:24Z" level=info msg="Dockie: Sync" file=" Jul 26 15:37:24.083324731 m/jkassis/[email protected]/logger.go: 192 " time="2021-07-26T15:37:26Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2063 became candidate" file=" Jul 26 15:37:26.990083654 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:26Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2063 received RequestVoteResp from n51496" file=" Jul 26 15:37:26.990177555 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:26Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2063 sent RequestVote to n01307" file=" Jul 26 15:37:26.990224095 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:26Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2063 sent RequestVote to n60202" file=" Jul 26 15:37:26.990262425 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:31Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2064 became candidate" file=" Jul 26 15:37:31.518545798 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:31Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2064 received RequestVoteResp from n51496" file=" Jul 26 15:37:31.518617848 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:31Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2064 sent RequestVote to n60202" file=" Jul 26 15:37:31.518678089 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:31Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2064 sent RequestVote to n01307" file=" Jul 26 15:37:31.518724599 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:34Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2065 became candidate" file=" Jul 26 15:37:34.955388054 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:34Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2065 received RequestVoteResp from n51496" file=" Jul 26 15:37:34.955460015 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:34Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2065 sent RequestVote to n60202" file=" Jul 26 15:37:34.955527906 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:34Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2065 sent RequestVote to n01307" file=" Jul 26 15:37:34.955571656 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:37Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2066 became candidate" file=" Jul 26 15:37:37.935998917 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:37Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2066 received RequestVoteResp from n51496" file=" Jul 26 15:37:37.936086478 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:37Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2066 sent RequestVote to n60202" file=" Jul 26 15:37:37.936130979 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:37Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2066 sent RequestVote to n01307" file=" Jul 26 15:37:37.936203939 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:41Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2067 became candidate" file=" Jul 26 15:37:41.716980849 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:41Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2067 received RequestVoteResp from n51496" file=" Jul 26 15:37:41.717060870 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:41Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2067 sent RequestVote to n01307" file=" Jul 26 15:37:41.717103741 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:41Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2067 sent RequestVote to n60202" file=" Jul 26 15:37:41.717138331 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:44Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2068 became candidate" file=" Jul 26 15:37:44.364128740 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:44Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2068 received RequestVoteResp from n51496" file=" Jul 26 15:37:44.364207420 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:44Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2068 sent RequestVote to n60202" file=" Jul 26 15:37:44.364263141 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:44Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2068 sent RequestVote to n01307" file=" Jul 26 15:37:44.364297501 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:49Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2069 became candidate" file=" Jul 26 15:37:49.054030614 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:49Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2069 received RequestVoteResp from n51496" file=" Jul 26 15:37:49.054098655 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:49Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2069 sent RequestVote to n60202" file=" Jul 26 15:37:49.054132365 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:49Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2069 sent RequestVote to n01307" file=" Jul 26 15:37:49.054156705 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:53Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2070 became candidate" file=" Jul 26 15:37:53.370702229 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:53Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2070 received RequestVoteResp from n51496" file=" Jul 26 15:37:53.370827060 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:53Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2070 sent RequestVote to n01307" file=" Jul 26 15:37:53.370877951 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:53Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2070 sent RequestVote to n60202" file=" Jul 26 15:37:53.370915571 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:56Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2071 became candidate" file=" Jul 26 15:37:56.124178702 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:56Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2071 received RequestVoteResp from n51496" file=" Jul 26 15:37:56.124251023 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:56Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2071 sent RequestVote to n60202" file=" Jul 26 15:37:56.124294673 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:56Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2071 sent RequestVote to n01307" file=" Jul 26 15:37:56.124328803 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:59Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2071 received RequestVoteResp from n60202" file=" Jul 26 15:37:59.455101132 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:37:59Z" level=warning msg="[f:8199,l:10533,t:2061,c:10532,a:10532] [00010:51496] t2071 received 2 votes and 0 rejections, quorum is 2" file=" Jul 26 15:37:59.455171182 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:38:35Z" level=warning msg="[f:8199,l:10534,t:2071,c:10534,a:10534] [00010:51496] t2071 received RequestVote with higher term (2072) from n01307" file=" Jul 26 15:38:35.114923022 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:38:35Z" level=warning msg="[f:8199,l:10534,t:2071,c:10534,a:10534] [00010:51496] t2071 become followerKE after receiving higher term from n01307" file=" Jul 26 15:38:35.114997792 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:38:35Z" level=warning msg="[f:8199,l:10534,t:2071,c:10534,a:10534] [00010:51496] t2072 cast vote from n01307 index 10534 term 2072, log term: 2071" file=" Jul 26 15:38:35.115051673 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:38:39Z" level=warning msg="[f:8199,l:10535,t:2072,c:10535,a:10535] [00010:51496] t2073 became candidate" file=" Jul 26 15:38:39.982614283 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:38:39Z" level=warning msg="[f:8199,l:10535,t:2072,c:10535,a:10535] [00010:51496] t2073 received RequestVoteResp from n51496" file=" Jul 26 15:38:39.982699724 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:38:39Z" level=warning msg="[f:8199,l:10535,t:2072,c:10535,a:10535] [00010:51496] t2073 sent RequestVote to n60202" file=" Jul 26 15:38:39.982769754 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:38:39Z" level=warning msg="[f:8199,l:10535,t:2072,c:10535,a:10535] [00010:51496] t2073 sent RequestVote to n01307" file=" Jul 26 15:38:39.982822345 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:38:40Z" level=warning msg="[f:8199,l:10535,t:2072,c:10535,a:10535] [00010:51496] t2073 received RequestVoteResp from n60202" file=" Jul 26 15:38:40.610936588 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:38:40Z" level=warning msg="[f:8199,l:10535,t:2072,c:10535,a:10535] [00010:51496] t2073 received 2 votes and 0 rejections, quorum is 2" file=" Jul 26 15:38:40.611002579 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:38:50Z" level=warning msg="[f:8199,l:10536,t:2073,c:10536,a:10536] [00010:51496] t2073 received RequestVote with higher term (2074) from n60202" file=" Jul 26 15:38:50.037728528 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:38:50Z" level=warning msg="[f:8199,l:10536,t:2073,c:10536,a:10536] [00010:51496] t2073 become followerKE after receiving higher term from n60202" file=" Jul 26 15:38:50.037819869 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:38:50Z" level=warning msg="[f:8199,l:10536,t:2073,c:10536,a:10536] [00010:51496] t2074 cast vote from n60202 index 10536 term 2074, log term: 2073" file=" Jul 26 15:38:50.037889089 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:39:05Z" level=warning msg="[f:8199,l:10537,t:2074,c:10537,a:10537] [00010:51496] t2075 became candidate" file=" Jul 26 15:39:05.601624660 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:39:05Z" level=warning msg="[f:8199,l:10537,t:2074,c:10537,a:10537] [00010:51496] t2075 received RequestVoteResp from n51496" file=" Jul 26 15:39:05.601696721 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:39:05Z" level=warning msg="[f:8199,l:10537,t:2074,c:10537,a:10537] [00010:51496] t2075 sent RequestVote to n60202" file=" Jul 26 15:39:05.601760811 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:39:05Z" level=warning msg="[f:8199,l:10537,t:2074,c:10537,a:10537] [00010:51496] t2075 sent RequestVote to n01307" file=" Jul 26 15:39:05.601799752 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:39:06Z" level=warning msg="[f:8199,l:10537,t:2074,c:10537,a:10537] [00010:51496] t2075 received RequestVoteResp from n01307" file=" Jul 26 15:39:06.226845042 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:39:06Z" level=warning msg="[f:8199,l:10537,t:2074,c:10537,a:10537] [00010:51496] t2075 received 2 votes and 0 rejections, quorum is 2" file=" Jul 26 15:39:06.226995434 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:39:16Z" level=warning msg="[f:8199,l:10538,t:2075,c:10538,a:10538] [00010:51496] t2075 received RequestVote with higher term (2076) from n60202" file=" Jul 26 15:39:16.356223207 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:39:16Z" level=warning msg="[f:8199,l:10538,t:2075,c:10538,a:10538] [00010:51496] t2075 become followerKE after receiving higher term from n60202" file=" Jul 26 15:39:16.356375918 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:39:16Z" level=warning msg="[f:8199,l:10538,t:2075,c:10538,a:10538] [00010:51496] t2076 cast vote from n60202 index 10538 term 2076, log term: 2075" file=" Jul 26 15:39:16.356451949 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:39:22Z" level=warning msg="[f:8199,l:10539,t:2076,c:10539,a:10539] [00010:51496] t2077 became candidate" file=" Jul 26 15:39:22.479956950 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:39:22Z" level=warning msg="[f:8199,l:10539,t:2076,c:10539,a:10539] [00010:51496] t2077 received RequestVoteResp from n51496" file=" Jul 26 15:39:22.480041451 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:39:22Z" level=warning msg="[f:8199,l:10539,t:2076,c:10539,a:10539] [00010:51496] t2077 sent RequestVote to n60202" file=" Jul 26 15:39:22.480087571 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:39:22Z" level=warning msg="[f:8199,l:10539,t:2076,c:10539,a:10539] [00010:51496] t2077 sent RequestVote to n01307" file=" Jul 26 15:39:22.480122571 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:39:22Z" level=warning msg="[f:8199,l:10539,t:2076,c:10539,a:10539] [00010:51496] t2077 received RequestVoteResp from n01307" file=" Jul 26 15:39:22.488323468 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:39:22Z" level=warning msg="[f:8199,l:10539,t:2076,c:10539,a:10539] [00010:51496] t2077 received 2 votes and 0 rejections, quorum is 2" file=" Jul 26 15:39:22.488421729 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:39:30Z" level=warning msg="[f:8199,l:10540,t:2077,c:10540,a:10540] [00010:51496] t2077 received RequestVote with higher term (2078) from n60202" file=" Jul 26 15:39:30.550826575 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:39:30Z" level=warning msg="[f:8199,l:10540,t:2077,c:10540,a:10540] [00010:51496] t2077 become followerKE after receiving higher term from n60202" file=" Jul 26 15:39:30.550898586 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:39:30Z" level=warning msg="[f:8199,l:10540,t:2077,c:10540,a:10540] [00010:51496] t2078 cast vote from n60202 index 10540 term 2078, log term: 2077" file=" Jul 26 15:39:30.550947366 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:39:35Z" level=warning msg="[f:8199,l:10541,t:2078,c:10541,a:10541] [00010:51496] t2079 became candidate" file=" Jul 26 15:39:35.202893180 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:39:35Z" level=warning msg="[f:8199,l:10541,t:2078,c:10541,a:10541] [00010:51496] t2079 received RequestVoteResp from n51496" file=" Jul 26 15:39:35.203010911 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:39:35Z" level=warning msg="[f:8199,l:10541,t:2078,c:10541,a:10541] [00010:51496] t2079 sent RequestVote to n60202" file=" Jul 26 15:39:35.203054091 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:39:35Z" level=warning msg="[f:8199,l:10541,t:2078,c:10541,a:10541] [00010:51496] t2079 sent RequestVote to n01307" file=" Jul 26 15:39:35.203088752 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:39:35Z" level=warning msg="[f:8199,l:10541,t:2078,c:10541,a:10541] [00010:51496] t2079 received RequestVoteResp from n01307" file=" Jul 26 15:39:35.212605670 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:39:35Z" level=warning msg="[f:8199,l:10541,t:2078,c:10541,a:10541] [00010:51496] t2079 received 2 votes and 0 rejections, quorum is 2" file=" Jul 26 15:39:35.212675841 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:39:39Z" level=warning msg="[f:8199,l:10542,t:2079,c:10542,a:10542] [00010:51496] t2079 received RequestVote with higher term (2080) from n60202" file=" Jul 26 15:39:39.180233316 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:39:39Z" level=warning msg="[f:8199,l:10542,t:2079,c:10542,a:10542] [00010:51496] t2079 become followerKE after receiving higher term from n60202" file=" Jul 26 15:39:39.180296707 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:39:39Z" level=warning msg="[f:8199,l:10542,t:2079,c:10542,a:10542] [00010:51496] t2080 cast vote from n60202 index 10542 term 2080, log term: 2079" file=" Jul 26 15:39:39.180331497 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:40:08Z" level=warning msg="[f:8199,l:10543,t:2080,c:10543,a:10543] [00010:51496] t2081 became candidate" file=" Jul 26 15:40:08.481454160 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:40:08Z" level=warning msg="[f:8199,l:10543,t:2080,c:10543,a:10543] [00010:51496] t2081 received RequestVoteResp from n51496" file=" Jul 26 15:40:08.481528661 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:40:08Z" level=warning msg="[f:8199,l:10543,t:2080,c:10543,a:10543] [00010:51496] t2081 sent RequestVote to n01307" file=" Jul 26 15:40:08.481575112 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:40:08Z" level=warning msg="[f:8199,l:10543,t:2080,c:10543,a:10543] [00010:51496] t2081 sent RequestVote to n60202" file=" Jul 26 15:40:08.481637962 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:40:08Z" level=warning msg="[f:8199,l:10543,t:2080,c:10543,a:10543] [00010:51496] t2081 received RequestVoteResp from n01307" file=" Jul 26 15:40:08.907860510 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:40:08Z" level=warning msg="[f:8199,l:10543,t:2080,c:10543,a:10543] [00010:51496] t2081 received 2 votes and 0 rejections, quorum is 2" file=" Jul 26 15:40:08.907921871 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:40:21Z" level=warning msg="[f:8199,l:10544,t:2081,c:10544,a:10544] [00010:51496] t2081 received RequestVote with higher term (2082) from n60202" file=" Jul 26 15:40:21.749203483 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:40:21Z" level=warning msg="[f:8199,l:10544,t:2081,c:10544,a:10544] [00010:51496] t2081 become followerKE after receiving higher term from n60202" file=" Jul 26 15:40:21.749278664 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:40:21Z" level=warning msg="[f:8199,l:10544,t:2081,c:10544,a:10544] [00010:51496] t2082 cast vote from n60202 index 10544 term 2082, log term: 2081" file=" Jul 26 15:40:21.749333335 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:40:38Z" level=warning msg="[f:8199,l:10545,t:2082,c:10545,a:10545] [00010:51496] t2083 became candidate" file=" Jul 26 15:40:38.753088828 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:40:38Z" level=warning msg="[f:8199,l:10545,t:2082,c:10545,a:10545] [00010:51496] t2083 received RequestVoteResp from n51496" file=" Jul 26 15:40:38.753170768 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:40:38Z" level=warning msg="[f:8199,l:10545,t:2082,c:10545,a:10545] [00010:51496] t2083 sent RequestVote to n60202" file=" Jul 26 15:40:38.753217269 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:40:38Z" level=warning msg="[f:8199,l:10545,t:2082,c:10545,a:10545] [00010:51496] t2083 sent RequestVote to n01307" file=" Jul 26 15:40:38.753252059 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:40:38Z" level=warning msg="[f:8199,l:10545,t:2082,c:10545,a:10545] [00010:51496] t2083 received RequestVoteResp from n01307" file=" Jul 26 15:40:38.760483609 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:40:38Z" level=warning msg="[f:8199,l:10545,t:2082,c:10545,a:10545] [00010:51496] t2083 received 2 votes and 0 rejections, quorum is 2" file=" Jul 26 15:40:38.760558879 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:40:46Z" level=warning msg="[f:8199,l:10546,t:2083,c:10546,a:10546] [00010:51496] t2083 received RequestVote with higher term (2084) from n60202" file=" Jul 26 15:40:46.836300509 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:40:46Z" level=warning msg="[f:8199,l:10546,t:2083,c:10546,a:10546] [00010:51496] t2083 become followerKE after receiving higher term from n60202" file=" Jul 26 15:40:46.836360629 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:40:46Z" level=warning msg="[f:8199,l:10546,t:2083,c:10546,a:10546] [00010:51496] t2084 cast vote from n60202 index 10546 term 2084, log term: 2083" file=" Jul 26 15:40:46.836408690 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:41:00Z" level=info msg="Dockie: Sync" file=" Jul 26 15:41:00.452406150 m/jkassis/[email protected]/logger.go: 192 " time="2021-07-26T15:41:06Z" level=warning msg="[f:8199,l:10547,t:2084,c:10547,a:10547] [00010:51496] t2085 became candidate" file=" Jul 26 15:41:06.929914215 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:41:06Z" level=warning msg="[f:8199,l:10547,t:2084,c:10547,a:10547] [00010:51496] t2085 received RequestVoteResp from n51496" file=" Jul 26 15:41:06.929990595 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:41:06Z" level=warning msg="[f:8199,l:10547,t:2084,c:10547,a:10547] [00010:51496] t2085 sent RequestVote to n60202" file=" Jul 26 15:41:06.930036696 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:41:06Z" level=warning msg="[f:8199,l:10547,t:2084,c:10547,a:10547] [00010:51496] t2085 sent RequestVote to n01307" file=" Jul 26 15:41:06.930073416 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:41:06Z" level=warning msg="[f:8199,l:10547,t:2084,c:10547,a:10547] [00010:51496] t2085 received RequestVoteResp from n01307" file=" Jul 26 15:41:06.939327798 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:41:06Z" level=warning msg="[f:8199,l:10547,t:2084,c:10547,a:10547] [00010:51496] t2085 received 2 votes and 0 rejections, quorum is 2" file=" Jul 26 15:41:06.939413549 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:41:29Z" level=warning msg="[f:8199,l:10548,t:2085,c:10548,a:10548] [00010:51496] t2085 received RequestVote with higher term (2086) from n60202" file=" Jul 26 15:41:29.013683701 om/jkassis/[email protected]/entry.go: 313 " time="2021-07-26T15:41:29Z" level=warning msg="[f:8199,l:10548,t:2085,c:10548,a:10548] [00010:51496] t2085 become followerKE after receiving higher term from n60202" file=" Jul 26 15:41:29.013799602 om/jkassis/[email protected]/entry.go: 313 "

Steps to reproduce the behavior

Not sure. Hoping the logs will shed some light on it.

Jul 26 '21 15:07 jkassis

@jkassis thanks for reporting the issue.

based on the log above, it seems to me that there are other raft nodes (n60202) probably running on different machines, any chance you can provide their log of the same time period as well? thanks.

Jul 27 '21 01:07 lni

@jkassis I had another look at the log you provided above, you changed the verbosity level to WARNING for the raft package. Any chance you can set it to INFO, that is to leave the verbosity level to its default, and restart your system? Together with the logs from nodes within the same affected cluster, that would help to identify the issue.

It would also be useful if you could provide the Config.ElectionRTT and NodeHostConfig.RTTMillisecond settings.

If possible, could you please email me at [email protected], I have a few questions to ask. I couldn't find your email address on your github profile. Thanks.

Jul 28 '21 02:07 lni

hi @lni. i'll do everything i can. i might still have the logs for those other nodes. give me til tomorrow am to produce those.

Jul 28 '21 18:07 jkassis

Here is an instance of this from today (with warning level):

time="2021-07-28T15:25:45Z" level=info msg="Dockie: Sync" file=" Jul 28 15:25:45.056150409 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:29:17Z" level=info msg="Dockie: Sync" file=" Jul 28 15:29:17.094191847 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:32:49Z" level=info msg="Dockie: Sync" file=" Jul 28 15:32:49.428926003 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:36:22Z" level=info msg="Dockie: Sync" file=" Jul 28 15:36:22.265593685 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:39:54Z" level=info msg="Dockie: Sync" file=" Jul 28 15:39:54.532167537 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:43:26Z" level=info msg="Dockie: Sync" file=" Jul 28 15:43:26.332010502 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:46:58Z" level=info msg="Dockie: Sync" file=" Jul 28 15:46:58.202359241 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:50:30Z" level=info msg="Dockie: Sync" file=" Jul 28 15:50:30.626288594 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:51:50Z" level=warning msg="[f:8202,l:10736,t:2366,c:10736,a:10736] [00010:60202] t2367 became candidate" file=" Jul 28 15:51:50.795805496 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:51:50Z" level=warning msg="[f:8202,l:10736,t:2366,c:10736,a:10736] [00010:60202] t2367 received RequestVoteResp from n60202" file=" Jul 28 15:51:50.795895397 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:51:50Z" level=warning msg="[f:8202,l:10736,t:2366,c:10736,a:10736] [00010:60202] t2367 sent RequestVote to n51496" file=" Jul 28 15:51:50.795941358 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:51:50Z" level=warning msg="[f:8202,l:10736,t:2366,c:10736,a:10736] [00010:60202] t2367 sent RequestVote to n01307" file=" Jul 28 15:51:50.795976978 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:51:50Z" level=warning msg="[f:8202,l:10736,t:2366,c:10736,a:10736] [00010:60202] t2367 received RequestVoteResp from n01307" file=" Jul 28 15:51:50.810204789 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:51:50Z" level=warning msg="[f:8202,l:10736,t:2366,c:10736,a:10736] [00010:60202] t2367 received 2 votes and 0 rejections, quorum is 2" file=" Jul 28 15:51:50.810276600 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:14Z" level=warning msg="[f:8202,l:10737,t:2367,c:10737,a:10737] [00010:60202] t2367 received RequestVote with higher term (2368) from n51496" file=" Jul 28 15:52:14.489437372 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:14Z" level=warning msg="[f:8202,l:10737,t:2367,c:10737,a:10737] [00010:60202] t2367 become followerKE after receiving higher term from n51496" file=" Jul 28 15:52:14.489503993 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:14Z" level=warning msg="[f:8202,l:10737,t:2367,c:10737,a:10737] [00010:60202] t2368 cast vote from n51496 index 10737 term 2368, log term: 2367" file=" Jul 28 15:52:14.489559444 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:14Z" level=warning msg="[f:8202,l:10737,t:2367,c:10737,a:10737] [00010:60202] t2368 received RequestVote with higher term (2369) from n51496" file=" Jul 28 15:52:14.489600374 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:14Z" level=warning msg="[f:8202,l:10737,t:2367,c:10737,a:10737] [00010:60202] t2368 become followerKE after receiving higher term from n51496" file=" Jul 28 15:52:14.489631805 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:14Z" level=warning msg="[f:8202,l:10737,t:2367,c:10737,a:10737] [00010:60202] t2369 cast vote from n51496 index 10737 term 2369, log term: 2367" file=" Jul 28 15:52:14.489668155 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:14Z" level=warning msg="[f:8202,l:10737,t:2367,c:10737,a:10737] [00010:60202] t2369 received RequestVote with higher term (2370) from n51496" file=" Jul 28 15:52:14.489722906 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:14Z" level=warning msg="[f:8202,l:10737,t:2367,c:10737,a:10737] [00010:60202] t2369 become followerKE after receiving higher term from n51496" file=" Jul 28 15:52:14.489756146 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:14Z" level=warning msg="[f:8202,l:10737,t:2367,c:10737,a:10737] [00010:60202] t2370 cast vote from n51496 index 10737 term 2370, log term: 2367" file=" Jul 28 15:52:14.489790027 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:14Z" level=warning msg="[f:8202,l:10737,t:2367,c:10737,a:10737] [00010:60202] t2370 received RequestVote with higher term (2371) from n51496" file=" Jul 28 15:52:14.489822687 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:14Z" level=warning msg="[f:8202,l:10737,t:2367,c:10737,a:10737] [00010:60202] t2370 become followerKE after receiving higher term from n51496" file=" Jul 28 15:52:14.489859008 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:14Z" level=warning msg="[f:8202,l:10737,t:2367,c:10737,a:10737] [00010:60202] t2371 cast vote from n51496 index 10737 term 2371, log term: 2367" file=" Jul 28 15:52:14.489897078 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:14Z" level=warning msg="[f:8202,l:10737,t:2367,c:10737,a:10737] [00010:60202] t2371 received RequestVote with higher term (2372) from n51496" file=" Jul 28 15:52:14.489938699 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:14Z" level=warning msg="[f:8202,l:10737,t:2367,c:10737,a:10737] [00010:60202] t2371 become followerKE after receiving higher term from n51496" file=" Jul 28 15:52:14.489968139 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:14Z" level=warning msg="[f:8202,l:10737,t:2367,c:10737,a:10737] [00010:60202] t2372 cast vote from n51496 index 10737 term 2372, log term: 2367" file=" Jul 28 15:52:14.490005869 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:14Z" level=warning msg="[f:8202,l:10737,t:2367,c:10737,a:10737] [00010:60202] t2372 received RequestVote with higher term (2373) from n51496" file=" Jul 28 15:52:14.490040570 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:14Z" level=warning msg="[f:8202,l:10737,t:2367,c:10737,a:10737] [00010:60202] t2372 become followerKE after receiving higher term from n51496" file=" Jul 28 15:52:14.490072560 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:14Z" level=warning msg="[f:8202,l:10737,t:2367,c:10737,a:10737] [00010:60202] t2373 cast vote from n51496 index 10737 term 2373, log term: 2367" file=" Jul 28 15:52:14.490111401 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:28Z" level=warning msg="[f:8202,l:10738,t:2373,c:10737,a:10737] [00010:60202] t2374 became candidate" file=" Jul 28 15:52:28.970354198 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:28Z" level=warning msg="[f:8202,l:10738,t:2373,c:10737,a:10737] [00010:60202] t2374 received RequestVoteResp from n60202" file=" Jul 28 15:52:28.970444769 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:28Z" level=warning msg="[f:8202,l:10738,t:2373,c:10737,a:10737] [00010:60202] t2374 sent RequestVote to n51496" file=" Jul 28 15:52:28.970489589 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:28Z" level=warning msg="[f:8202,l:10738,t:2373,c:10737,a:10737] [00010:60202] t2374 sent RequestVote to n01307" file=" Jul 28 15:52:28.970525310 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:28Z" level=warning msg="[f:8202,l:10738,t:2373,c:10737,a:10737] [00010:60202] t2374 received RequestVoteResp from n01307" file=" Jul 28 15:52:28.978486456 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:28Z" level=warning msg="[f:8202,l:10738,t:2373,c:10737,a:10737] [00010:60202] t2374 received 2 votes and 0 rejections, quorum is 2" file=" Jul 28 15:52:28.978551796 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:57Z" level=error msg="send batch failed, target [00010:51496] (write tcp 10.128.241.218:44212->10.128.236.169:9999: i/o timeout), 1" file=" Jul 28 15:52:57.459155374 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:57Z" level=warning msg="breaker dockie-0.dockie-int.fg.svc.cluster.local:9999 to dockie-2.dockie-int.fg.svc.cluster.local:9999 failed, connect and process failed: write tcp 10.128.241.218:44212->10.128.236.169:9999: i/o timeout" file=" Jul 28 15:52:57.459365027 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:57Z" level=warning msg="dockie-2.dockie-int.fg.svc.cluster.local:9999 became unreachable, affecting 1 raft nodes" file=" Jul 28 15:52:57.459470798 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:02Z" level=error msg="Nodehost dockie-0.dockie-int.fg.svc.cluster.local:9999 failed to get a connection to dockie-2.dockie-int.fg.svc.cluster.local:9999, dial tcp 10.128.236.169:9999: i/o timeout" file=" Jul 28 15:53:02.995174452 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:02Z" level=warning msg="breaker dockie-0.dockie-int.fg.svc.cluster.local:9999 to dockie-2.dockie-int.fg.svc.cluster.local:9999 failed, connect and process failed: dial tcp 10.128.236.169:9999: i/o timeout" file=" Jul 28 15:53:02.995251683 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:02Z" level=warning msg="dockie-2.dockie-int.fg.svc.cluster.local:9999 became unreachable, affecting 1 raft nodes" file=" Jul 28 15:53:02.995291423 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:08Z" level=error msg="Nodehost dockie-0.dockie-int.fg.svc.cluster.local:9999 failed to get a connection to dockie-2.dockie-int.fg.svc.cluster.local:9999, dial tcp 10.128.236.169:9999: i/o timeout" file=" Jul 28 15:53:08.780763223 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:08Z" level=warning msg="breaker dockie-0.dockie-int.fg.svc.cluster.local:9999 to dockie-2.dockie-int.fg.svc.cluster.local:9999 failed, connect and process failed: dial tcp 10.128.236.169:9999: i/o timeout" file=" Jul 28 15:53:08.780836444 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:08Z" level=warning msg="dockie-2.dockie-int.fg.svc.cluster.local:9999 became unreachable, affecting 1 raft nodes" file=" Jul 28 15:53:08.780877555 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:02Z" level=info msg="Dockie: Sync" file=" Jul 28 15:54:02.844001758 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:54:07Z" level=error msg="send batch failed, target [00010:51496] (write tcp 10.128.241.218:35660->10.128.236.169:9999: i/o timeout), 1" file=" Jul 28 15:54:07.254305125 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:07Z" level=warning msg="breaker dockie-0.dockie-int.fg.svc.cluster.local:9999 to dockie-2.dockie-int.fg.svc.cluster.local:9999 failed, connect and process failed: write tcp 10.128.241.218:35660->10.128.236.169:9999: i/o timeout" file=" Jul 28 15:54:07.254477967 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:07Z" level=warning msg="dockie-2.dockie-int.fg.svc.cluster.local:9999 became unreachable, affecting 1 raft nodes" file=" Jul 28 15:54:07.254524667 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2374 received RequestVote with higher term (2375) from n51496" file=" Jul 28 15:54:35.802486963 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2374 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.802554833 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2375 rejected vote n51496 index10738 term2375,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.802615004 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2375 received RequestVote with higher term (2376) from n51496" file=" Jul 28 15:54:35.802681765 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2375 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.802820897 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2376 rejected vote n51496 index10738 term2376,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.802880917 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2376 received RequestVote with higher term (2377) from n51496" file=" Jul 28 15:54:35.802917818 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2376 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.802949548 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2377 rejected vote n51496 index10738 term2377,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.802990069 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2377 received RequestVote with higher term (2378) from n51496" file=" Jul 28 15:54:35.803025629 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2377 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.803059630 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2378 rejected vote n51496 index10738 term2378,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.803100610 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2378 received RequestVote with higher term (2379) from n51496" file=" Jul 28 15:54:35.803139340 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2378 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.803172031 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2379 rejected vote n51496 index10738 term2379,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.803212011 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2379 received RequestVote with higher term (2380) from n51496" file=" Jul 28 15:54:35.803247982 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2379 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.803282842 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2380 rejected vote n51496 index10738 term2380,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.803328143 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2380 received RequestVote with higher term (2381) from n51496" file=" Jul 28 15:54:35.807094198 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2380 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.807154129 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2381 rejected vote n51496 index10738 term2381,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.807200310 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2381 received RequestVote with higher term (2382) from n51496" file=" Jul 28 15:54:35.807240440 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2381 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.807278351 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2382 rejected vote n51496 index10738 term2382,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.807318361 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2382 received RequestVote with higher term (2383) from n51496" file=" Jul 28 15:54:35.807359092 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2382 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.807390462 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2383 rejected vote n51496 index10738 term2383,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.807451533 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2383 received RequestVote with higher term (2384) from n51496" file=" Jul 28 15:54:35.807505803 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2383 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.807539364 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2384 rejected vote n51496 index10738 term2384,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.807614875 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2384 received RequestVote with higher term (2385) from n51496" file=" Jul 28 15:54:35.807677185 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2384 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.807730606 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2385 rejected vote n51496 index10738 term2385,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.807781787 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2385 received RequestVote with higher term (2386) from n51496" file=" Jul 28 15:54:35.807865188 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2385 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.807896888 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2386 rejected vote n51496 index10738 term2386,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.807941859 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2386 received RequestVote with higher term (2387) from n51496" file=" Jul 28 15:54:35.808006369 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2386 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.808040950 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2387 rejected vote n51496 index10738 term2387,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.808126631 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2387 received RequestVote with higher term (2388) from n51496" file=" Jul 28 15:54:35.808209222 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2387 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.808241792 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2388 rejected vote n51496 index10738 term2388,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.808289793 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2388 received RequestVote with higher term (2389) from n51496" file=" Jul 28 15:54:35.808359434 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2388 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.808393294 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2389 rejected vote n51496 index10738 term2389,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.808450285 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2389 received RequestVote with higher term (2390) from n51496" file=" Jul 28 15:54:35.808489785 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2389 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.808547046 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2390 rejected vote n51496 index10738 term2390,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.808587376 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2390 received RequestVote with higher term (2391) from n51496" file=" Jul 28 15:54:35.808638617 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2390 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.810876314 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2391 rejected vote n51496 index10738 term2391,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.810957765 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2391 received RequestVote with higher term (2392) from n51496" file=" Jul 28 15:54:35.811004526 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2391 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.811047666 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2392 rejected vote n51496 index10738 term2392,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.811093577 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2392 received RequestVote with higher term (2393) from n51496" file=" Jul 28 15:54:35.811131797 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2392 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.811168128 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2393 rejected vote n51496 index10738 term2393,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.811211858 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2393 received RequestVote with higher term (2394) from n51496" file=" Jul 28 15:54:35.811250849 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2393 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.811286429 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2394 rejected vote n51496 index10738 term2394,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.811332510 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2394 received RequestVote with higher term (2395) from n51496" file=" Jul 28 15:54:35.811371870 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2394 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.811405981 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2395 rejected vote n51496 index10738 term2395,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.811476121 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2395 received RequestVote with higher term (2396) from n51496" file=" Jul 28 15:54:35.811519192 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2395 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.811555542 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2396 rejected vote n51496 index10738 term2396,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.811599073 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:37Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2397 became candidate" file=" Jul 28 15:54:37.466996300 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:37Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2397 received RequestVoteResp from n60202" file=" Jul 28 15:54:37.467069461 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:37Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2397 sent RequestVote to n01307" file=" Jul 28 15:54:37.467113692 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:37Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2397 sent RequestVote to n51496" file=" Jul 28 15:54:37.467149382 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:37Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2397 received RequestVoteResp from n51496" file=" Jul 28 15:54:37.473325867 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:37Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:60202] t2397 received 2 votes and 0 rejections, quorum is 2" file=" Jul 28 15:54:37.473497069 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:57:35Z" level=info msg="Dockie: Sync" file=" Jul 28 15:57:35.435954105 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T16:01:06Z" level=info msg="Dockie: Sync" file=" Jul 28 16:01:06.826935856 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T16:04:36Z" level=info msg="Dockie: Sync" file=" Jul 28 16:04:36.856901957 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T16:08:08Z" level=info msg="Dockie: Sync" file=" Jul 28 16:08:08.086261750 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T16:11:40Z" level=info msg="Dockie: Sync" file=" Jul 28 16:11:40.059204170 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T16:15:12Z" level=info msg="Dockie: Sync" file=" Jul 28 16:15:12.045271591 m/jkassis/[email protected]/logger.go: 192 "

Jul 29 '21 02:07 jkassis

Node 2:

time="2021-07-28T14:58:35Z" level=info msg="Dockie: Sync" file=" Jul 28 14:58:35.899452963 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:02:07Z" level=info msg="Dockie: Sync" file=" Jul 28 15:02:07.940046780 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:05:38Z" level=info msg="Dockie: Sync" file=" Jul 28 15:05:38.886741626 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:09:10Z" level=info msg="Dockie: Sync" file=" Jul 28 15:09:10.924764679 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:12:42Z" level=info msg="Dockie: Sync" file=" Jul 28 15:12:42.356378586 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:16:14Z" level=info msg="Dockie: Sync" file=" Jul 28 15:16:14.390855839 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:19:46Z" level=info msg="Dockie: Sync" file=" Jul 28 15:19:46.981519709 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:20:16Z" level=warning msg="[f:8202,l:10724,t:2333,c:10724,a:10724] [00010:01307] t2333 received RequestVote with higher term (2335) from n60202" file=" Jul 28 15:20:16.592170369 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:16Z" level=warning msg="[f:8202,l:10724,t:2333,c:10724,a:10724] [00010:01307] t2333 become followerKE after receiving higher term from n60202" file=" Jul 28 15:20:16.592282331 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:16Z" level=warning msg="[f:8202,l:10724,t:2333,c:10724,a:10724] [00010:01307] t2335 cast vote from n60202 index 10724 term 2335, log term: 2333" file=" Jul 28 15:20:16.592327131 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:34Z" level=warning msg="[f:8202,l:10725,t:2335,c:10725,a:10725] [00010:01307] t2335 received RequestVote with higher term (2341) from n51496" file=" Jul 28 15:20:34.776674615 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:34Z" level=warning msg="[f:8202,l:10725,t:2335,c:10725,a:10725] [00010:01307] t2335 become followerKE after receiving higher term from n51496" file=" Jul 28 15:20:34.776736846 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:34Z" level=warning msg="[f:8202,l:10725,t:2335,c:10725,a:10725] [00010:01307] t2341 rejected vote n51496 index10724 term2341,logterm2333,granttrue,utdfalse" file=" Jul 28 15:20:34.776784097 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:37Z" level=warning msg="[f:8202,l:10725,t:2335,c:10725,a:10725] [00010:01307] t2341 received RequestVote with higher term (2342) from n51496" file=" Jul 28 15:20:37.007583735 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:37Z" level=warning msg="[f:8202,l:10725,t:2335,c:10725,a:10725] [00010:01307] t2341 become followerKE after receiving higher term from n51496" file=" Jul 28 15:20:37.007687746 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:37Z" level=warning msg="[f:8202,l:10725,t:2335,c:10725,a:10725] [00010:01307] t2342 rejected vote n51496 index10724 term2342,logterm2333,granttrue,utdfalse" file=" Jul 28 15:20:37.007747287 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:39Z" level=warning msg="[f:8202,l:10725,t:2335,c:10725,a:10725] [00010:01307] t2343 became candidate" file=" Jul 28 15:20:39.495752249 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:39Z" level=warning msg="[f:8202,l:10725,t:2335,c:10725,a:10725] [00010:01307] t2343 received RequestVoteResp from n01307" file=" Jul 28 15:20:39.495834540 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:39Z" level=warning msg="[f:8202,l:10725,t:2335,c:10725,a:10725] [00010:01307] t2343 sent RequestVote to n51496" file=" Jul 28 15:20:39.495882430 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:39Z" level=warning msg="[f:8202,l:10725,t:2335,c:10725,a:10725] [00010:01307] t2343 sent RequestVote to n60202" file=" Jul 28 15:20:39.495917491 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:39Z" level=warning msg="[f:8202,l:10725,t:2335,c:10725,a:10725] [00010:01307] t2343 received RequestVoteResp from n60202" file=" Jul 28 15:20:39.502479774 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:39Z" level=warning msg="[f:8202,l:10725,t:2335,c:10725,a:10725] [00010:01307] t2343 received 2 votes and 0 rejections, quorum is 2" file=" Jul 28 15:20:39.502572945 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:42Z" level=warning msg="[f:8202,l:10726,t:2343,c:10726,a:10726] [00010:01307] t2343 rejected vote n51496 index10724 term2343,logterm2333,grantfalse,utdfalse" file=" Jul 28 15:20:42.263413512 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:45Z" level=warning msg="[f:8202,l:10726,t:2343,c:10726,a:10726] [00010:01307] t2343 received RequestVote with higher term (2344) from n51496" file=" Jul 28 15:20:45.528313550 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:45Z" level=warning msg="[f:8202,l:10726,t:2343,c:10726,a:10726] [00010:01307] t2343 become followerKE after receiving higher term from n51496" file=" Jul 28 15:20:45.528487722 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:45Z" level=warning msg="[f:8202,l:10726,t:2343,c:10726,a:10726] [00010:01307] t2344 rejected vote n51496 index10724 term2344,logterm2333,granttrue,utdfalse" file=" Jul 28 15:20:45.528545563 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:46Z" level=warning msg="[f:8202,l:10726,t:2343,c:10726,a:10726] [00010:01307] t2345 became candidate" file=" Jul 28 15:20:46.711853671 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:46Z" level=warning msg="[f:8202,l:10726,t:2343,c:10726,a:10726] [00010:01307] t2345 received RequestVoteResp from n01307" file=" Jul 28 15:20:46.711953933 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:46Z" level=warning msg="[f:8202,l:10726,t:2343,c:10726,a:10726] [00010:01307] t2345 sent RequestVote to n51496" file=" Jul 28 15:20:46.712004973 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:46Z" level=warning msg="[f:8202,l:10726,t:2343,c:10726,a:10726] [00010:01307] t2345 sent RequestVote to n60202" file=" Jul 28 15:20:46.712041064 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:46Z" level=warning msg="[f:8202,l:10726,t:2343,c:10726,a:10726] [00010:01307] t2345 received RequestVoteResp from n60202" file=" Jul 28 15:20:46.718797919 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:46Z" level=warning msg="[f:8202,l:10726,t:2343,c:10726,a:10726] [00010:01307] t2345 received 2 votes and 0 rejections, quorum is 2" file=" Jul 28 15:20:46.718888671 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:47Z" level=warning msg="[f:8202,l:10727,t:2345,c:10727,a:10727] [00010:01307] t2345 rejected vote n51496 index10724 term2345,logterm2333,grantfalse,utdfalse" file=" Jul 28 15:20:47.396625286 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:57Z" level=warning msg="[f:8202,l:10727,t:2345,c:10727,a:10727] [00010:01307] t2345 received RequestVote with higher term (2346) from n51496" file=" Jul 28 15:20:57.010549952 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:57Z" level=warning msg="[f:8202,l:10727,t:2345,c:10727,a:10727] [00010:01307] t2345 become followerKE after receiving higher term from n51496" file=" Jul 28 15:20:57.010623863 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:57Z" level=warning msg="[f:8202,l:10727,t:2345,c:10727,a:10727] [00010:01307] t2346 cast vote from n51496 index 10727 term 2346, log term: 2345" file=" Jul 28 15:20:57.010678034 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:00Z" level=warning msg="[f:8202,l:10727,t:2345,c:10727,a:10727] [00010:01307] t2347 became candidate" file=" Jul 28 15:21:00.246360029 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:00Z" level=warning msg="[f:8202,l:10727,t:2345,c:10727,a:10727] [00010:01307] t2347 received RequestVoteResp from n01307" file=" Jul 28 15:21:00.246453890 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:00Z" level=warning msg="[f:8202,l:10727,t:2345,c:10727,a:10727] [00010:01307] t2347 sent RequestVote to n51496" file=" Jul 28 15:21:00.246498991 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:00Z" level=warning msg="[f:8202,l:10727,t:2345,c:10727,a:10727] [00010:01307] t2347 sent RequestVote to n60202" file=" Jul 28 15:21:00.246534831 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:00Z" level=warning msg="[f:8202,l:10727,t:2345,c:10727,a:10727] [00010:01307] t2347 received RequestVoteResp from n60202" file=" Jul 28 15:21:00.253118696 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:00Z" level=warning msg="[f:8202,l:10727,t:2345,c:10727,a:10727] [00010:01307] t2347 received 2 votes and 0 rejections, quorum is 2" file=" Jul 28 15:21:00.253226667 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:01Z" level=warning msg="[f:8202,l:10728,t:2347,c:10728,a:10728] [00010:01307] t2347 rejected vote n51496 index10727 term2347,logterm2345,grantfalse,utdfalse" file=" Jul 28 15:21:01.042016544 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:05Z" level=warning msg="[f:8202,l:10728,t:2347,c:10728,a:10728] [00010:01307] t2347 received RequestVote with higher term (2348) from n51496" file=" Jul 28 15:21:05.487462572 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:05Z" level=warning msg="[f:8202,l:10728,t:2347,c:10728,a:10728] [00010:01307] t2347 become followerKE after receiving higher term from n51496" file=" Jul 28 15:21:05.487527873 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:05Z" level=warning msg="[f:8202,l:10728,t:2347,c:10728,a:10728] [00010:01307] t2348 rejected vote n51496 index10727 term2348,logterm2345,granttrue,utdfalse" file=" Jul 28 15:21:05.487581374 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:09Z" level=warning msg="[f:8202,l:10728,t:2347,c:10728,a:10728] [00010:01307] t2348 cast vote from n60202 index 10728 term 2348, log term: 2347" file=" Jul 28 15:21:09.299068175 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:09Z" level=warning msg="[f:8202,l:10729,t:2348,c:10729,a:10729] [00010:01307] t2348 received RequestVote with higher term (2349) from n51496" file=" Jul 28 15:21:09.944654870 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:09Z" level=warning msg="[f:8202,l:10729,t:2348,c:10729,a:10729] [00010:01307] t2348 become followerKE after receiving higher term from n51496" file=" Jul 28 15:21:09.944780872 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:09Z" level=warning msg="[f:8202,l:10729,t:2348,c:10729,a:10729] [00010:01307] t2349 rejected vote n51496 index10727 term2349,logterm2345,granttrue,utdfalse" file=" Jul 28 15:21:09.944837703 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:13Z" level=warning msg="[f:8202,l:10729,t:2348,c:10729,a:10729] [00010:01307] t2349 received RequestVote with higher term (2350) from n51496" file=" Jul 28 15:21:13.845300060 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:13Z" level=warning msg="[f:8202,l:10729,t:2348,c:10729,a:10729] [00010:01307] t2349 become followerKE after receiving higher term from n51496" file=" Jul 28 15:21:13.845376651 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:13Z" level=warning msg="[f:8202,l:10729,t:2348,c:10729,a:10729] [00010:01307] t2350 rejected vote n51496 index10727 term2350,logterm2345,granttrue,utdfalse" file=" Jul 28 15:21:13.845430492 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:14Z" level=warning msg="[f:8202,l:10729,t:2348,c:10729,a:10729] [00010:01307] t2351 became candidate" file=" Jul 28 15:21:14.174756077 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:14Z" level=warning msg="[f:8202,l:10729,t:2348,c:10729,a:10729] [00010:01307] t2351 received RequestVoteResp from n01307" file=" Jul 28 15:21:14.174825248 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:14Z" level=warning msg="[f:8202,l:10729,t:2348,c:10729,a:10729] [00010:01307] t2351 sent RequestVote to n60202" file=" Jul 28 15:21:14.174869279 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:14Z" level=warning msg="[f:8202,l:10729,t:2348,c:10729,a:10729] [00010:01307] t2351 sent RequestVote to n51496" file=" Jul 28 15:21:14.174907359 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:14Z" level=warning msg="[f:8202,l:10729,t:2348,c:10729,a:10729] [00010:01307] t2351 received RequestVoteResp from n60202" file=" Jul 28 15:21:14.181159561 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:14Z" level=warning msg="[f:8202,l:10729,t:2348,c:10729,a:10729] [00010:01307] t2351 received 2 votes and 0 rejections, quorum is 2" file=" Jul 28 15:21:14.181258042 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:18Z" level=warning msg="[f:8202,l:10730,t:2351,c:10730,a:10730] [00010:01307] t2351 rejected vote n51496 index10727 term2351,logterm2345,grantfalse,utdfalse" file=" Jul 28 15:21:18.503468277 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:22Z" level=warning msg="[f:8202,l:10730,t:2351,c:10730,a:10730] [00010:01307] t2351 received RequestVote with higher term (2352) from n51496" file=" Jul 28 15:21:22.248557614 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:22Z" level=warning msg="[f:8202,l:10730,t:2351,c:10730,a:10730] [00010:01307] t2351 become followerKE after receiving higher term from n51496" file=" Jul 28 15:21:22.248625615 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:22Z" level=warning msg="[f:8202,l:10730,t:2351,c:10730,a:10730] [00010:01307] t2352 rejected vote n51496 index10727 term2352,logterm2345,granttrue,utdfalse" file=" Jul 28 15:21:22.248683066 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:24Z" level=warning msg="[f:8202,l:10730,t:2351,c:10730,a:10730] [00010:01307] t2352 cast vote from n60202 index 10730 term 2352, log term: 2351" file=" Jul 28 15:21:24.994556283 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:25Z" level=warning msg="[f:8202,l:10731,t:2352,c:10731,a:10731] [00010:01307] t2352 received RequestVote with higher term (2353) from n51496" file=" Jul 28 15:21:25.244005140 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:25Z" level=warning msg="[f:8202,l:10731,t:2352,c:10731,a:10731] [00010:01307] t2352 become followerKE after receiving higher term from n51496" file=" Jul 28 15:21:25.244083641 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:25Z" level=warning msg="[f:8202,l:10731,t:2352,c:10731,a:10731] [00010:01307] t2353 rejected vote n51496 index10727 term2353,logterm2345,granttrue,utdfalse" file=" Jul 28 15:21:25.244141812 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:28Z" level=warning msg="[f:8202,l:10731,t:2352,c:10731,a:10731] [00010:01307] t2353 received RequestVote with higher term (2354) from n51496" file=" Jul 28 15:21:28.626660029 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:28Z" level=warning msg="[f:8202,l:10731,t:2352,c:10731,a:10731] [00010:01307] t2353 become followerKE after receiving higher term from n51496" file=" Jul 28 15:21:28.626729670 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:28Z" level=warning msg="[f:8202,l:10731,t:2352,c:10731,a:10731] [00010:01307] t2354 rejected vote n51496 index10727 term2354,logterm2345,granttrue,utdfalse" file=" Jul 28 15:21:28.626776550 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:29Z" level=warning msg="[f:8202,l:10731,t:2352,c:10731,a:10731] [00010:01307] t2355 became candidate" file=" Jul 28 15:21:29.438129576 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:29Z" level=warning msg="[f:8202,l:10731,t:2352,c:10731,a:10731] [00010:01307] t2355 received RequestVoteResp from n01307" file=" Jul 28 15:21:29.438226737 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:29Z" level=warning msg="[f:8202,l:10731,t:2352,c:10731,a:10731] [00010:01307] t2355 sent RequestVote to n60202" file=" Jul 28 15:21:29.438272438 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:29Z" level=warning msg="[f:8202,l:10731,t:2352,c:10731,a:10731] [00010:01307] t2355 sent RequestVote to n51496" file=" Jul 28 15:21:29.438309079 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:29Z" level=warning msg="[f:8202,l:10731,t:2352,c:10731,a:10731] [00010:01307] t2355 received RequestVoteResp from n60202" file=" Jul 28 15:21:29.444832475 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:29Z" level=warning msg="[f:8202,l:10731,t:2352,c:10731,a:10731] [00010:01307] t2355 received 2 votes and 0 rejections, quorum is 2" file=" Jul 28 15:21:29.444934457 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:45Z" level=warning msg="[f:8202,l:10732,t:2355,c:10732,a:10732] [00010:01307] t2355 received RequestVote with higher term (2356) from n51496" file=" Jul 28 15:21:45.503473100 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:45Z" level=warning msg="[f:8202,l:10732,t:2355,c:10732,a:10732] [00010:01307] t2355 become followerKE after receiving higher term from n51496" file=" Jul 28 15:21:45.503562011 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:45Z" level=warning msg="[f:8202,l:10732,t:2355,c:10732,a:10732] [00010:01307] t2356 cast vote from n51496 index 10732 term 2356, log term: 2355" file=" Jul 28 15:21:45.503617642 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:48Z" level=warning msg="[f:8202,l:10732,t:2355,c:10732,a:10732] [00010:01307] t2356 rejected vote n60202 index10732 term2356,logterm2355,grantfalse,utdtrue" file=" Jul 28 15:21:48.354651214 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:50Z" level=warning msg="[f:8202,l:10732,t:2355,c:10732,a:10732] [00010:01307] t2357 became candidate" file=" Jul 28 15:21:50.037916770 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:50Z" level=warning msg="[f:8202,l:10732,t:2355,c:10732,a:10732] [00010:01307] t2357 received RequestVoteResp from n01307" file=" Jul 28 15:21:50.038002861 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:50Z" level=warning msg="[f:8202,l:10732,t:2355,c:10732,a:10732] [00010:01307] t2357 sent RequestVote to n51496" file=" Jul 28 15:21:50.038049642 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:50Z" level=warning msg="[f:8202,l:10732,t:2355,c:10732,a:10732] [00010:01307] t2357 sent RequestVote to n60202" file=" Jul 28 15:21:50.038083532 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:50Z" level=warning msg="[f:8202,l:10732,t:2355,c:10732,a:10732] [00010:01307] t2357 received RequestVoteResp from n60202" file=" Jul 28 15:21:50.044970124 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:50Z" level=warning msg="[f:8202,l:10732,t:2355,c:10732,a:10732] [00010:01307] t2357 received 2 votes and 0 rejections, quorum is 2" file=" Jul 28 15:21:50.045044865 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:50Z" level=warning msg="[f:8202,l:10733,t:2357,c:10733,a:10733] [00010:01307] t2357 rejected vote n51496 index10732 term2357,logterm2355,grantfalse,utdfalse" file=" Jul 28 15:21:50.487440100 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:51Z" level=warning msg="[f:8202,l:10733,t:2357,c:10733,a:10733] [00010:01307] t2357 received RequestVote with higher term (2358) from n51496" file=" Jul 28 15:21:51.890343769 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:51Z" level=warning msg="[f:8202,l:10733,t:2357,c:10733,a:10733] [00010:01307] t2357 become followerKE after receiving higher term from n51496" file=" Jul 28 15:21:51.890406320 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:51Z" level=warning msg="[f:8202,l:10733,t:2357,c:10733,a:10733] [00010:01307] t2358 rejected vote n51496 index10732 term2358,logterm2355,granttrue,utdfalse" file=" Jul 28 15:21:51.890452991 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:52Z" level=warning msg="[f:8202,l:10733,t:2357,c:10733,a:10733] [00010:01307] t2359 became candidate" file=" Jul 28 15:21:52.719419234 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:52Z" level=warning msg="[f:8202,l:10733,t:2357,c:10733,a:10733] [00010:01307] t2359 received RequestVoteResp from n01307" file=" Jul 28 15:21:52.719494105 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:52Z" level=warning msg="[f:8202,l:10733,t:2357,c:10733,a:10733] [00010:01307] t2359 sent RequestVote to n51496" file=" Jul 28 15:21:52.719536616 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:52Z" level=warning msg="[f:8202,l:10733,t:2357,c:10733,a:10733] [00010:01307] t2359 sent RequestVote to n60202" file=" Jul 28 15:21:52.719572566 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:52Z" level=warning msg="[f:8202,l:10733,t:2357,c:10733,a:10733] [00010:01307] t2359 received RequestVoteResp from n60202" file=" Jul 28 15:21:52.726011601 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:52Z" level=warning msg="[f:8202,l:10733,t:2357,c:10733,a:10733] [00010:01307] t2359 received 2 votes and 0 rejections, quorum is 2" file=" Jul 28 15:21:52.726098322 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:02Z" level=warning msg="[f:8202,l:10734,t:2359,c:10734,a:10734] [00010:01307] t2359 received RequestVote with higher term (2360) from n51496" file=" Jul 28 15:22:02.279531357 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:02Z" level=warning msg="[f:8202,l:10734,t:2359,c:10734,a:10734] [00010:01307] t2359 become followerKE after receiving higher term from n51496" file=" Jul 28 15:22:02.279603878 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:02Z" level=warning msg="[f:8202,l:10734,t:2359,c:10734,a:10734] [00010:01307] t2360 cast vote from n51496 index 10734 term 2360, log term: 2359" file=" Jul 28 15:22:02.279659189 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:04Z" level=warning msg="[f:8202,l:10734,t:2359,c:10734,a:10734] [00010:01307] t2360 received RequestVote with higher term (2361) from n60202" file=" Jul 28 15:22:04.944149675 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:04Z" level=warning msg="[f:8202,l:10734,t:2359,c:10734,a:10734] [00010:01307] t2360 become followerKE after receiving higher term from n60202" file=" Jul 28 15:22:04.944226566 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:04Z" level=warning msg="[f:8202,l:10734,t:2359,c:10734,a:10734] [00010:01307] t2361 cast vote from n60202 index 10734 term 2361, log term: 2359" file=" Jul 28 15:22:04.944271807 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:28Z" level=warning msg="[f:8202,l:10735,t:2361,c:10735,a:10735] [00010:01307] t2361 received RequestVote with higher term (2362) from n51496" file=" Jul 28 15:22:28.120923088 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:28Z" level=warning msg="[f:8202,l:10735,t:2361,c:10735,a:10735] [00010:01307] t2361 become followerKE after receiving higher term from n51496" file=" Jul 28 15:22:28.121012659 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:28Z" level=warning msg="[f:8202,l:10735,t:2361,c:10735,a:10735] [00010:01307] t2362 cast vote from n51496 index 10735 term 2362, log term: 2361" file=" Jul 28 15:22:28.121085590 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:28Z" level=warning msg="[f:8202,l:10735,t:2361,c:10735,a:10735] [00010:01307] t2362 received RequestVote with higher term (2363) from n51496" file=" Jul 28 15:22:28.121123400 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:28Z" level=warning msg="[f:8202,l:10735,t:2361,c:10735,a:10735] [00010:01307] t2362 become followerKE after receiving higher term from n51496" file=" Jul 28 15:22:28.121176151 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:28Z" level=warning msg="[f:8202,l:10735,t:2361,c:10735,a:10735] [00010:01307] t2363 cast vote from n51496 index 10735 term 2363, log term: 2361" file=" Jul 28 15:22:28.121247692 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:28Z" level=warning msg="[f:8202,l:10735,t:2361,c:10735,a:10735] [00010:01307] t2363 received RequestVote with higher term (2364) from n51496" file=" Jul 28 15:22:28.123907361 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:28Z" level=warning msg="[f:8202,l:10735,t:2361,c:10735,a:10735] [00010:01307] t2363 become followerKE after receiving higher term from n51496" file=" Jul 28 15:22:28.123964222 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:28Z" level=warning msg="[f:8202,l:10735,t:2361,c:10735,a:10735] [00010:01307] t2364 cast vote from n51496 index 10735 term 2364, log term: 2361" file=" Jul 28 15:22:28.124005742 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:28Z" level=warning msg="[f:8202,l:10735,t:2361,c:10735,a:10735] [00010:01307] t2364 received RequestVote with higher term (2365) from n51496" file=" Jul 28 15:22:28.124046333 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:28Z" level=warning msg="[f:8202,l:10735,t:2361,c:10735,a:10735] [00010:01307] t2364 become followerKE after receiving higher term from n51496" file=" Jul 28 15:22:28.124078903 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:28Z" level=warning msg="[f:8202,l:10735,t:2361,c:10735,a:10735] [00010:01307] t2365 cast vote from n51496 index 10735 term 2365, log term: 2361" file=" Jul 28 15:22:28.124118244 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:28Z" level=warning msg="[f:8202,l:10735,t:2361,c:10735,a:10735] [00010:01307] t2365 received RequestVote with higher term (2366) from n51496" file=" Jul 28 15:22:28.124154514 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:28Z" level=warning msg="[f:8202,l:10735,t:2361,c:10735,a:10735] [00010:01307] t2365 become followerKE after receiving higher term from n51496" file=" Jul 28 15:22:28.124203855 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:28Z" level=warning msg="[f:8202,l:10735,t:2361,c:10735,a:10735] [00010:01307] t2366 cast vote from n51496 index 10735 term 2366, log term: 2361" file=" Jul 28 15:22:28.124251766 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:23:20Z" level=info msg="Dockie: Sync" file=" Jul 28 15:23:20.020683598 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:26:52Z" level=info msg="Dockie: Sync" file=" Jul 28 15:26:52.096401226 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:30:24Z" level=info msg="Dockie: Sync" file=" Jul 28 15:30:24.948051984 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:33:56Z" level=info msg="Dockie: Sync" file=" Jul 28 15:33:56.840155842 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:37:28Z" level=info msg="Dockie: Sync" file=" Jul 28 15:37:28.721810217 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:41:00Z" level=info msg="Dockie: Sync" file=" Jul 28 15:41:00.038407279 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:44:31Z" level=info msg="Dockie: Sync" file=" Jul 28 15:44:31.435588721 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:48:03Z" level=info msg="Dockie: Sync" file=" Jul 28 15:48:03.165834180 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:51:34Z" level=info msg="Dockie: Sync" file=" Jul 28 15:51:34.968903417 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:51:50Z" level=warning msg="[f:8202,l:10736,t:2366,c:10736,a:10736] [00010:01307] t2366 received RequestVote with higher term (2367) from n60202" file=" Jul 28 15:51:50.804913407 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:51:50Z" level=warning msg="[f:8202,l:10736,t:2366,c:10736,a:10736] [00010:01307] t2366 become followerKE after receiving higher term from n60202" file=" Jul 28 15:51:50.805049529 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:51:50Z" level=warning msg="[f:8202,l:10736,t:2366,c:10736,a:10736] [00010:01307] t2367 cast vote from n60202 index 10736 term 2367, log term: 2366" file=" Jul 28 15:51:50.805107940 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:14Z" level=warning msg="[f:8202,l:10737,t:2367,c:10737,a:10737] [00010:01307] t2367 received RequestVote with higher term (2368) from n51496" file=" Jul 28 15:52:14.488871862 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:14Z" level=warning msg="[f:8202,l:10737,t:2367,c:10737,a:10737] [00010:01307] t2367 become followerKE after receiving higher term from n51496" file=" Jul 28 15:52:14.488940793 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:14Z" level=warning msg="[f:8202,l:10737,t:2367,c:10737,a:10737] [00010:01307] t2368 cast vote from n51496 index 10737 term 2368, log term: 2367" file=" Jul 28 15:52:14.489007974 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:14Z" level=warning msg="[f:8202,l:10737,t:2367,c:10737,a:10737] [00010:01307] t2368 received RequestVote with higher term (2369) from n51496" file=" Jul 28 15:52:14.489046644 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:14Z" level=warning msg="[f:8202,l:10737,t:2367,c:10737,a:10737] [00010:01307] t2368 become followerKE after receiving higher term from n51496" file=" Jul 28 15:52:14.489080845 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:14Z" level=warning msg="[f:8202,l:10737,t:2367,c:10737,a:10737] [00010:01307] t2369 cast vote from n51496 index 10737 term 2369, log term: 2367" file=" Jul 28 15:52:14.489123325 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:14Z" level=warning msg="[f:8202,l:10737,t:2367,c:10737,a:10737] [00010:01307] t2369 received RequestVote with higher term (2370) from n51496" file=" Jul 28 15:52:14.489162896 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:14Z" level=warning msg="[f:8202,l:10737,t:2367,c:10737,a:10737] [00010:01307] t2369 become followerKE after receiving higher term from n51496" file=" Jul 28 15:52:14.490659516 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:14Z" level=warning msg="[f:8202,l:10737,t:2367,c:10737,a:10737] [00010:01307] t2370 cast vote from n51496 index 10737 term 2370, log term: 2367" file=" Jul 28 15:52:14.490730457 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:14Z" level=warning msg="[f:8202,l:10737,t:2367,c:10737,a:10737] [00010:01307] t2370 received RequestVote with higher term (2371) from n51496" file=" Jul 28 15:52:14.490784498 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:14Z" level=warning msg="[f:8202,l:10737,t:2367,c:10737,a:10737] [00010:01307] t2370 become followerKE after receiving higher term from n51496" file=" Jul 28 15:52:14.490816488 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:14Z" level=warning msg="[f:8202,l:10737,t:2367,c:10737,a:10737] [00010:01307] t2371 cast vote from n51496 index 10737 term 2371, log term: 2367" file=" Jul 28 15:52:14.490867279 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:14Z" level=warning msg="[f:8202,l:10737,t:2367,c:10737,a:10737] [00010:01307] t2371 received RequestVote with higher term (2372) from n51496" file=" Jul 28 15:52:14.490910929 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:14Z" level=warning msg="[f:8202,l:10737,t:2367,c:10737,a:10737] [00010:01307] t2371 become followerKE after receiving higher term from n51496" file=" Jul 28 15:52:14.490946610 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:14Z" level=warning msg="[f:8202,l:10737,t:2367,c:10737,a:10737] [00010:01307] t2372 cast vote from n51496 index 10737 term 2372, log term: 2367" file=" Jul 28 15:52:14.490998340 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:14Z" level=warning msg="[f:8202,l:10737,t:2367,c:10737,a:10737] [00010:01307] t2372 received RequestVote with higher term (2373) from n51496" file=" Jul 28 15:52:14.491043101 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:14Z" level=warning msg="[f:8202,l:10737,t:2367,c:10737,a:10737] [00010:01307] t2372 become followerKE after receiving higher term from n51496" file=" Jul 28 15:52:14.491082742 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:14Z" level=warning msg="[f:8202,l:10737,t:2367,c:10737,a:10737] [00010:01307] t2373 cast vote from n51496 index 10737 term 2373, log term: 2367" file=" Jul 28 15:52:14.491126022 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:28Z" level=warning msg="[f:8202,l:10738,t:2373,c:10738,a:10738] [00010:01307] t2373 received RequestVote with higher term (2374) from n60202" file=" Jul 28 15:52:28.975163229 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:28Z" level=warning msg="[f:8202,l:10738,t:2373,c:10738,a:10738] [00010:01307] t2373 become followerKE after receiving higher term from n60202" file=" Jul 28 15:52:28.975296611 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:28Z" level=warning msg="[f:8202,l:10738,t:2373,c:10738,a:10738] [00010:01307] t2374 cast vote from n60202 index 10738 term 2374, log term: 2373" file=" Jul 28 15:52:28.975344802 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2374 received RequestVote with higher term (2375) from n51496" file=" Jul 28 15:54:35.805620490 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2374 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.805681551 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2375 rejected vote n51496 index10738 term2375,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.805733212 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2375 received RequestVote with higher term (2376) from n51496" file=" Jul 28 15:54:35.805772893 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2375 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.805827073 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2376 rejected vote n51496 index10738 term2376,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.805868014 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2376 received RequestVote with higher term (2377) from n51496" file=" Jul 28 15:54:35.809159762 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2376 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.809243233 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2377 rejected vote n51496 index10738 term2377,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.809297354 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2377 received RequestVote with higher term (2378) from n51496" file=" Jul 28 15:54:35.809347795 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2377 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.809377915 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2378 rejected vote n51496 index10738 term2378,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.809411516 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2378 received RequestVote with higher term (2379) from n51496" file=" Jul 28 15:54:35.809449546 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2378 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.809491727 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2379 rejected vote n51496 index10738 term2379,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.809535898 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2379 received RequestVote with higher term (2380) from n51496" file=" Jul 28 15:54:35.809578718 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2379 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.809615209 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2380 rejected vote n51496 index10738 term2380,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.809658680 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2380 received RequestVote with higher term (2381) from n51496" file=" Jul 28 15:54:35.809701140 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2380 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.809736111 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2381 rejected vote n51496 index10738 term2381,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.809778641 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2381 received RequestVote with higher term (2382) from n51496" file=" Jul 28 15:54:35.809834722 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2381 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.809869893 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2382 rejected vote n51496 index10738 term2382,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.809919073 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2382 received RequestVote with higher term (2383) from n51496" file=" Jul 28 15:54:35.809979794 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2382 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.810022645 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2383 rejected vote n51496 index10738 term2383,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.810093186 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2383 received RequestVote with higher term (2384) from n51496" file=" Jul 28 15:54:35.810127916 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2383 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.810160727 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2384 rejected vote n51496 index10738 term2384,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.812746175 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2384 received RequestVote with higher term (2385) from n51496" file=" Jul 28 15:54:35.812826246 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2384 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.812858026 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2385 rejected vote n51496 index10738 term2385,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.812898337 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2385 received RequestVote with higher term (2386) from n51496" file=" Jul 28 15:54:35.812948188 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2385 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.812978848 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2386 rejected vote n51496 index10738 term2386,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.813025519 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2386 received RequestVote with higher term (2387) from n51496" file=" Jul 28 15:54:35.813070220 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2386 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.813109900 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2387 rejected vote n51496 index10738 term2387,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.813156581 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2387 received RequestVote with higher term (2388) from n51496" file=" Jul 28 15:54:35.813969793 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2387 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.814034664 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2388 rejected vote n51496 index10738 term2388,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.814070514 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2388 received RequestVote with higher term (2389) from n51496" file=" Jul 28 15:54:35.814100875 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2388 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.814125005 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2389 rejected vote n51496 index10738 term2389,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.814154765 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2389 received RequestVote with higher term (2390) from n51496" file=" Jul 28 15:54:35.814180846 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2389 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.814309378 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2390 rejected vote n51496 index10738 term2390,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.814349088 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2390 received RequestVote with higher term (2391) from n51496" file=" Jul 28 15:54:35.814379259 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2390 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.814404209 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2391 rejected vote n51496 index10738 term2391,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.814438000 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2391 received RequestVote with higher term (2392) from n51496" file=" Jul 28 15:54:35.814466400 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2391 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.814490870 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2392 rejected vote n51496 index10738 term2392,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.814523901 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2392 received RequestVote with higher term (2393) from n51496" file=" Jul 28 15:54:35.814556731 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2392 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.814597392 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2393 rejected vote n51496 index10738 term2393,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.814632952 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2393 received RequestVote with higher term (2394) from n51496" file=" Jul 28 15:54:35.814659343 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2393 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.814688533 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2394 rejected vote n51496 index10738 term2394,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.814737604 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2394 received RequestVote with higher term (2395) from n51496" file=" Jul 28 15:54:35.814779555 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2394 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.814804425 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2395 rejected vote n51496 index10738 term2395,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.814837825 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2395 received RequestVote with higher term (2396) from n51496" file=" Jul 28 15:54:35.814864086 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2395 become followerKE after receiving higher term from n51496" file=" Jul 28 15:54:35.814891176 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2396 rejected vote n51496 index10738 term2396,logterm2373,granttrue,utdfalse" file=" Jul 28 15:54:35.814923377 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:37Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2396 received RequestVote with higher term (2397) from n60202" file=" Jul 28 15:54:37.469903809 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:37Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2396 become followerKE after receiving higher term from n60202" file=" Jul 28 15:54:37.469960720 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:37Z" level=warning msg="[f:8202,l:10739,t:2374,c:10739,a:10739] [00010:01307] t2397 cast vote from n60202 index 10739 term 2397, log term: 2374" file=" Jul 28 15:54:37.470009371 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:55:07Z" level=info msg="Dockie: Sync" file=" Jul 28 15:55:07.190807949 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:58:38Z" level=info msg="Dockie: Sync" file=" Jul 28 15:58:38.772398934 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T16:02:10Z" level=info msg="Dockie: Sync" file=" Jul 28 16:02:10.303837297 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T16:05:42Z" level=info msg="Dockie: Sync" file=" Jul 28 16:05:42.161794435 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T16:09:14Z" level=info msg="Dockie: Sync" file=" Jul 28 16:09:14.421179190 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T16:12:46Z" level=info msg="Dockie: Sync" file=" Jul 28 16:12:46.175910507 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T16:16:17Z" level=info msg="Dockie: Sync" file=" Jul 28 16:16:17.949329755 m/jkassis/[email protected]/logger.go: 192 "

Jul 29 '21 02:07 jkassis

Node 3

time="2021-07-28T14:44:28Z" level=info msg="Dockie: Sync" file=" Jul 28 14:44:28.919821436 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T14:48:04Z" level=info msg="Dockie: Sync" file=" Jul 28 14:48:04.925211766 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T14:51:41Z" level=info msg="Dockie: Sync" file=" Jul 28 14:51:41.188625161 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T14:55:18Z" level=info msg="Dockie: Sync" file=" Jul 28 14:55:18.503380797 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T14:58:55Z" level=info msg="Dockie: Sync" file=" Jul 28 14:58:55.923715440 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:02:33Z" level=info msg="Dockie: Sync" file=" Jul 28 15:02:33.590079412 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:06:12Z" level=info msg="Dockie: Sync" file=" Jul 28 15:06:12.377654233 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:09:49Z" level=info msg="Dockie: Sync" file=" Jul 28 15:09:49.222907389 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:13:27Z" level=info msg="Dockie: Sync" file=" Jul 28 15:13:27.203813410 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:17:06Z" level=info msg="Dockie: Sync" file=" Jul 28 15:17:06.027780461 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:20:13Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2334 became candidate" file=" Jul 28 15:20:13.048460822 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:13Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2334 received RequestVoteResp from n51496" file=" Jul 28 15:20:13.048541863 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:13Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2334 sent RequestVote to n60202" file=" Jul 28 15:20:13.048590263 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:13Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2334 sent RequestVote to n01307" file=" Jul 28 15:20:13.048626294 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:16Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2335 became candidate" file=" Jul 28 15:20:16.735877046 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:16Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2335 received RequestVoteResp from n51496" file=" Jul 28 15:20:16.735959807 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:16Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2335 sent RequestVote to n60202" file=" Jul 28 15:20:16.736009448 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:16Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2335 sent RequestVote to n01307" file=" Jul 28 15:20:16.736044898 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:18Z" level=error msg="Nodehost dockie-2.dockie-int.fg.svc.cluster.local:9999 failed to get a connection to dockie-1.dockie-int.fg.svc.cluster.local:9999, dial tcp 10.130.87.232:9999: i/o timeout" file=" Jul 28 15:20:18.051898225 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:18Z" level=warning msg="breaker dockie-2.dockie-int.fg.svc.cluster.local:9999 to dockie-1.dockie-int.fg.svc.cluster.local:9999 failed, connect and process failed: dial tcp 10.130.87.232:9999: i/o timeout" file=" Jul 28 15:20:18.051970226 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:18Z" level=warning msg="dockie-1.dockie-int.fg.svc.cluster.local:9999 became unreachable, affecting 1 raft nodes" file=" Jul 28 15:20:18.052010226 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:19Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2336 became candidate" file=" Jul 28 15:20:19.024188093 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:19Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2336 received RequestVoteResp from n51496" file=" Jul 28 15:20:19.024257964 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:19Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2336 sent RequestVote to n60202" file=" Jul 28 15:20:19.024301345 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:19Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2336 sent RequestVote to n01307" file=" Jul 28 15:20:19.024337005 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:21Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2337 became candidate" file=" Jul 28 15:20:21.532880193 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:21Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2337 received RequestVoteResp from n51496" file=" Jul 28 15:20:21.532986434 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:21Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2337 sent RequestVote to n60202" file=" Jul 28 15:20:21.533029885 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:21Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2337 sent RequestVote to n01307" file=" Jul 28 15:20:21.533115226 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:24Z" level=error msg="Nodehost dockie-2.dockie-int.fg.svc.cluster.local:9999 failed to get a connection to dockie-1.dockie-int.fg.svc.cluster.local:9999, dial tcp 10.130.87.232:9999: i/o timeout" file=" Jul 28 15:20:24.027483730 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:24Z" level=warning msg="breaker dockie-2.dockie-int.fg.svc.cluster.local:9999 to dockie-1.dockie-int.fg.svc.cluster.local:9999 failed, connect and process failed: dial tcp 10.130.87.232:9999: i/o timeout" file=" Jul 28 15:20:24.027560881 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:24Z" level=warning msg="dockie-1.dockie-int.fg.svc.cluster.local:9999 became unreachable, affecting 1 raft nodes" file=" Jul 28 15:20:24.027619692 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:24Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2338 became candidate" file=" Jul 28 15:20:24.510061392 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:24Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2338 received RequestVoteResp from n51496" file=" Jul 28 15:20:24.510133033 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:24Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2338 sent RequestVote to n60202" file=" Jul 28 15:20:24.510177004 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:24Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2338 sent RequestVote to n01307" file=" Jul 28 15:20:24.510212184 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:27Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2339 became candidate" file=" Jul 28 15:20:27.437675202 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:27Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2339 received RequestVoteResp from n51496" file=" Jul 28 15:20:27.437760643 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:27Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2339 sent RequestVote to n60202" file=" Jul 28 15:20:27.437806943 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:27Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2339 sent RequestVote to n01307" file=" Jul 28 15:20:27.437842014 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:31Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2340 became candidate" file=" Jul 28 15:20:31.233252168 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:31Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2340 received RequestVoteResp from n51496" file=" Jul 28 15:20:31.233340419 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:31Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2340 sent RequestVote to n01307" file=" Jul 28 15:20:31.233386979 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:31Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2340 sent RequestVote to n60202" file=" Jul 28 15:20:31.233422140 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:32Z" level=error msg="Nodehost dockie-2.dockie-int.fg.svc.cluster.local:9999 failed to get a connection to dockie-1.dockie-int.fg.svc.cluster.local:9999, dial tcp 10.130.87.232:9999: i/o timeout" file=" Jul 28 15:20:32.440975902 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:32Z" level=warning msg="breaker dockie-2.dockie-int.fg.svc.cluster.local:9999 to dockie-1.dockie-int.fg.svc.cluster.local:9999 failed, connect and process failed: dial tcp 10.130.87.232:9999: i/o timeout" file=" Jul 28 15:20:32.441047023 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:32Z" level=warning msg="dockie-1.dockie-int.fg.svc.cluster.local:9999 became unreachable, affecting 1 raft nodes" file=" Jul 28 15:20:32.441089124 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:33Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2341 became candidate" file=" Jul 28 15:20:33.713827519 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:33Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2341 received RequestVoteResp from n51496" file=" Jul 28 15:20:33.713900440 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:33Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2341 sent RequestVote to n60202" file=" Jul 28 15:20:33.713945001 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:33Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2341 sent RequestVote to n01307" file=" Jul 28 15:20:33.713982091 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:34Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2341 received RequestVoteResp rejection from n01307" file=" Jul 28 15:20:34.782719554 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:34Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2341 received 1 votes and 1 rejections, quorum is 2" file=" Jul 28 15:20:34.782838546 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:36Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2342 became candidate" file=" Jul 28 15:20:36.389605846 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:36Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2342 received RequestVoteResp from n51496" file=" Jul 28 15:20:36.389679536 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:36Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2342 sent RequestVote to n60202" file=" Jul 28 15:20:36.389727017 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:36Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2342 sent RequestVote to n01307" file=" Jul 28 15:20:36.389788278 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:37Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2342 received RequestVoteResp rejection from n01307" file=" Jul 28 15:20:37.010935897 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:37Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2342 received 1 votes and 1 rejections, quorum is 2" file=" Jul 28 15:20:37.011042658 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:39Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2343 became candidate" file=" Jul 28 15:20:39.121307757 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:39Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2343 received RequestVoteResp from n51496" file=" Jul 28 15:20:39.121388828 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:39Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2343 sent RequestVote to n60202" file=" Jul 28 15:20:39.121436328 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:39Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2343 sent RequestVote to n01307" file=" Jul 28 15:20:39.121474519 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:42Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2344 became candidate" file=" Jul 28 15:20:42.640945956 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:42Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2344 received RequestVoteResp from n51496" file=" Jul 28 15:20:42.641052857 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:42Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2344 sent RequestVote to n60202" file=" Jul 28 15:20:42.641099828 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:42Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2344 sent RequestVote to n01307" file=" Jul 28 15:20:42.641132678 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:42Z" level=info msg="Dockie: Sync" file=" Jul 28 15:20:42.643651086 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:20:47Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2345 became candidate" file=" Jul 28 15:20:47.384994681 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:47Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2345 received RequestVoteResp from n51496" file=" Jul 28 15:20:47.385064141 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:47Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2345 sent RequestVote to n01307" file=" Jul 28 15:20:47.385109592 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:47Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2345 sent RequestVote to n60202" file=" Jul 28 15:20:47.385148642 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:49Z" level=warning msg="[f:8199,l:10724,t:2333,c:10724,a:10724] [00010:51496] t2345 rejected vote n01307 index10726 term2345,logterm2343,grantfalse,utdtrue" file=" Jul 28 15:20:49.004940754 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:57Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2346 became candidate" file=" Jul 28 15:20:57.006164300 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:57Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2346 received RequestVoteResp from n51496" file=" Jul 28 15:20:57.006254431 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:57Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2346 sent RequestVote to n60202" file=" Jul 28 15:20:57.006297731 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:20:57Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2346 sent RequestVote to n01307" file=" Jul 28 15:20:57.006333952 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:00Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2347 became candidate" file=" Jul 28 15:21:00.418895276 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:00Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2347 received RequestVoteResp from n51496" file=" Jul 28 15:21:00.418968737 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:00Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2347 sent RequestVote to n60202" file=" Jul 28 15:21:00.419012387 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:00Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2347 sent RequestVote to n01307" file=" Jul 28 15:21:00.419046717 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:05Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2348 became candidate" file=" Jul 28 15:21:05.268377293 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:05Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2348 received RequestVoteResp from n51496" file=" Jul 28 15:21:05.268449404 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:05Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2348 sent RequestVote to n60202" file=" Jul 28 15:21:05.268495994 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:05Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2348 sent RequestVote to n01307" file=" Jul 28 15:21:05.268535735 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:06Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2348 received RequestVoteResp rejection from n01307" file=" Jul 28 15:21:06.933988466 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:06Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2348 received 1 votes and 1 rejections, quorum is 2" file=" Jul 28 15:21:06.934059137 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:09Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2349 became candidate" file=" Jul 28 15:21:09.732401362 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:09Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2349 received RequestVoteResp from n51496" file=" Jul 28 15:21:09.732491783 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:09Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2349 sent RequestVote to n60202" file=" Jul 28 15:21:09.732537513 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:09Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2349 sent RequestVote to n01307" file=" Jul 28 15:21:09.732591474 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:13Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2350 became candidate" file=" Jul 28 15:21:13.835841202 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:13Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2350 received RequestVoteResp from n51496" file=" Jul 28 15:21:13.835917463 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:13Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2350 sent RequestVote to n60202" file=" Jul 28 15:21:13.835964564 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:13Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2350 sent RequestVote to n01307" file=" Jul 28 15:21:13.836002384 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:18Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2351 became candidate" file=" Jul 28 15:21:18.285876615 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:18Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2351 received RequestVoteResp from n51496" file=" Jul 28 15:21:18.285952526 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:18Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2351 sent RequestVote to n01307" file=" Jul 28 15:21:18.285996727 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:18Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2351 sent RequestVote to n60202" file=" Jul 28 15:21:18.286032287 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:21Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2352 became candidate" file=" Jul 28 15:21:21.621828661 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:21Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2352 received RequestVoteResp from n51496" file=" Jul 28 15:21:21.621903151 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:21Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2352 sent RequestVote to n60202" file=" Jul 28 15:21:21.621947792 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:21Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2352 sent RequestVote to n01307" file=" Jul 28 15:21:21.621982762 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:22Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2352 received RequestVoteResp rejection from n01307" file=" Jul 28 15:21:22.797970188 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:22Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2352 received 1 votes and 1 rejections, quorum is 2" file=" Jul 28 15:21:22.798031229 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:25Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2353 became candidate" file=" Jul 28 15:21:25.239798516 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:25Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2353 received RequestVoteResp from n51496" file=" Jul 28 15:21:25.239872997 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:25Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2353 sent RequestVote to n60202" file=" Jul 28 15:21:25.239918428 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:25Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2353 sent RequestVote to n01307" file=" Jul 28 15:21:25.239952098 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:25Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2353 received RequestVoteResp rejection from n01307" file=" Jul 28 15:21:25.248193629 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:25Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2353 received 1 votes and 1 rejections, quorum is 2" file=" Jul 28 15:21:25.248261290 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:28Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2354 became candidate" file=" Jul 28 15:21:28.622255556 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:28Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2354 received RequestVoteResp from n51496" file=" Jul 28 15:21:28.622326517 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:28Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2354 sent RequestVote to n60202" file=" Jul 28 15:21:28.622374248 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:28Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2354 sent RequestVote to n01307" file=" Jul 28 15:21:28.622409538 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:28Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2354 received RequestVoteResp rejection from n01307" file=" Jul 28 15:21:28.629727749 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:28Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2354 received 1 votes and 1 rejections, quorum is 2" file=" Jul 28 15:21:28.629853941 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:29Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2354 received RequestVote with higher term (2355) from n01307" file=" Jul 28 15:21:29.441892862 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:29Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2354 become followerKE after receiving higher term from n01307" file=" Jul 28 15:21:29.441965203 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:29Z" level=warning msg="[f:8199,l:10727,t:2345,c:10727,a:10727] [00010:51496] t2355 cast vote from n01307 index 10731 term 2355, log term: 2352" file=" Jul 28 15:21:29.442020114 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:44Z" level=warning msg="[f:8199,l:10732,t:2355,c:10732,a:10732] [00010:51496] t2356 became candidate" file=" Jul 28 15:21:44.885506750 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:44Z" level=warning msg="[f:8199,l:10732,t:2355,c:10732,a:10732] [00010:51496] t2356 received RequestVoteResp from n51496" file=" Jul 28 15:21:44.885604821 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:44Z" level=warning msg="[f:8199,l:10732,t:2355,c:10732,a:10732] [00010:51496] t2356 sent RequestVote to n60202" file=" Jul 28 15:21:44.885651272 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:44Z" level=warning msg="[f:8199,l:10732,t:2355,c:10732,a:10732] [00010:51496] t2356 sent RequestVote to n01307" file=" Jul 28 15:21:44.885682472 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:48Z" level=warning msg="[f:8199,l:10732,t:2355,c:10732,a:10732] [00010:51496] t2356 rejected vote n60202 index10732 term2356,logterm2355,grantfalse,utdtrue" file=" Jul 28 15:21:48.564172376 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:49Z" level=warning msg="[f:8199,l:10732,t:2355,c:10732,a:10732] [00010:51496] t2357 became candidate" file=" Jul 28 15:21:49.023527562 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:49Z" level=warning msg="[f:8199,l:10732,t:2355,c:10732,a:10732] [00010:51496] t2357 received RequestVoteResp from n51496" file=" Jul 28 15:21:49.023598172 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:49Z" level=warning msg="[f:8199,l:10732,t:2355,c:10732,a:10732] [00010:51496] t2357 sent RequestVote to n01307" file=" Jul 28 15:21:49.023642683 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:49Z" level=warning msg="[f:8199,l:10732,t:2355,c:10732,a:10732] [00010:51496] t2357 sent RequestVote to n60202" file=" Jul 28 15:21:49.023680303 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:51Z" level=warning msg="[f:8199,l:10732,t:2355,c:10732,a:10732] [00010:51496] t2358 became candidate" file=" Jul 28 15:21:51.882080002 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:51Z" level=warning msg="[f:8199,l:10732,t:2355,c:10732,a:10732] [00010:51496] t2358 received RequestVoteResp from n51496" file=" Jul 28 15:21:51.882153913 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:51Z" level=warning msg="[f:8199,l:10732,t:2355,c:10732,a:10732] [00010:51496] t2358 sent RequestVote to n60202" file=" Jul 28 15:21:51.882198604 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:51Z" level=warning msg="[f:8199,l:10732,t:2355,c:10732,a:10732] [00010:51496] t2358 sent RequestVote to n01307" file=" Jul 28 15:21:51.882234044 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:54Z" level=warning msg="[f:8199,l:10732,t:2355,c:10732,a:10732] [00010:51496] t2358 received RequestVoteResp rejection from n01307" file=" Jul 28 15:21:54.030112777 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:54Z" level=warning msg="[f:8199,l:10732,t:2355,c:10732,a:10732] [00010:51496] t2358 received 1 votes and 1 rejections, quorum is 2" file=" Jul 28 15:21:54.030179857 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:54Z" level=warning msg="[f:8199,l:10732,t:2355,c:10732,a:10732] [00010:51496] t2358 received RequestVote with higher term (2359) from n01307" file=" Jul 28 15:21:54.030221518 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:54Z" level=warning msg="[f:8199,l:10732,t:2355,c:10732,a:10732] [00010:51496] t2358 become followerKE after receiving higher term from n01307" file=" Jul 28 15:21:54.030256868 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:21:54Z" level=warning msg="[f:8199,l:10732,t:2355,c:10732,a:10732] [00010:51496] t2359 cast vote from n01307 index 10733 term 2359, log term: 2357" file=" Jul 28 15:21:54.030310439 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:01Z" level=warning msg="[f:8199,l:10734,t:2359,c:10734,a:10734] [00010:51496] t2360 became candidate" file=" Jul 28 15:22:01.660822856 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:01Z" level=warning msg="[f:8199,l:10734,t:2359,c:10734,a:10734] [00010:51496] t2360 received RequestVoteResp from n51496" file=" Jul 28 15:22:01.660899137 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:01Z" level=warning msg="[f:8199,l:10734,t:2359,c:10734,a:10734] [00010:51496] t2360 sent RequestVote to n60202" file=" Jul 28 15:22:01.660945387 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:01Z" level=warning msg="[f:8199,l:10734,t:2359,c:10734,a:10734] [00010:51496] t2360 sent RequestVote to n01307" file=" Jul 28 15:22:01.661007478 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:02Z" level=warning msg="[f:8199,l:10734,t:2359,c:10734,a:10734] [00010:51496] t2360 received RequestVoteResp from n60202" file=" Jul 28 15:22:02.492172593 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:02Z" level=warning msg="[f:8199,l:10734,t:2359,c:10734,a:10734] [00010:51496] t2360 received 2 votes and 0 rejections, quorum is 2" file=" Jul 28 15:22:02.492255404 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:08Z" level=warning msg="[f:8199,l:10735,t:2360,c:10734,a:10734] [00010:51496] t2360 received RequestVote with higher term (2361) from n60202" file=" Jul 28 15:22:08.053284816 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:08Z" level=warning msg="[f:8199,l:10735,t:2360,c:10734,a:10734] [00010:51496] t2360 become followerKE after receiving higher term from n60202" file=" Jul 28 15:22:08.053352817 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:08Z" level=warning msg="[f:8199,l:10735,t:2360,c:10734,a:10734] [00010:51496] t2361 rejected vote n60202 index10734 term2361,logterm2359,granttrue,utdfalse" file=" Jul 28 15:22:08.053420388 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:14Z" level=warning msg="[f:8199,l:10735,t:2361,c:10734,a:10734] [00010:51496] t2362 became candidate" file=" Jul 28 15:22:14.372589917 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:14Z" level=warning msg="[f:8199,l:10735,t:2361,c:10734,a:10734] [00010:51496] t2362 received RequestVoteResp from n51496" file=" Jul 28 15:22:14.372677858 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:14Z" level=warning msg="[f:8199,l:10735,t:2361,c:10734,a:10734] [00010:51496] t2362 sent RequestVote to n60202" file=" Jul 28 15:22:14.372720958 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:14Z" level=warning msg="[f:8199,l:10735,t:2361,c:10734,a:10734] [00010:51496] t2362 sent RequestVote to n01307" file=" Jul 28 15:22:14.372787359 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:18Z" level=warning msg="[f:8199,l:10735,t:2361,c:10734,a:10734] [00010:51496] t2363 became candidate" file=" Jul 28 15:22:18.199998489 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:18Z" level=warning msg="[f:8199,l:10735,t:2361,c:10734,a:10734] [00010:51496] t2363 received RequestVoteResp from n51496" file=" Jul 28 15:22:18.200071590 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:18Z" level=warning msg="[f:8199,l:10735,t:2361,c:10734,a:10734] [00010:51496] t2363 sent RequestVote to n60202" file=" Jul 28 15:22:18.200119160 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:18Z" level=warning msg="[f:8199,l:10735,t:2361,c:10734,a:10734] [00010:51496] t2363 sent RequestVote to n01307" file=" Jul 28 15:22:18.200155270 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:20Z" level=warning msg="[f:8199,l:10735,t:2361,c:10734,a:10734] [00010:51496] t2364 became candidate" file=" Jul 28 15:22:20.602878424 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:20Z" level=warning msg="[f:8199,l:10735,t:2361,c:10734,a:10734] [00010:51496] t2364 received RequestVoteResp from n51496" file=" Jul 28 15:22:20.602965215 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:20Z" level=warning msg="[f:8199,l:10735,t:2361,c:10734,a:10734] [00010:51496] t2364 sent RequestVote to n60202" file=" Jul 28 15:22:20.603008516 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:20Z" level=warning msg="[f:8199,l:10735,t:2361,c:10734,a:10734] [00010:51496] t2364 sent RequestVote to n01307" file=" Jul 28 15:22:20.603043186 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:24Z" level=warning msg="[f:8199,l:10735,t:2361,c:10734,a:10734] [00010:51496] t2365 became candidate" file=" Jul 28 15:22:24.728145608 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:24Z" level=warning msg="[f:8199,l:10735,t:2361,c:10734,a:10734] [00010:51496] t2365 received RequestVoteResp from n51496" file=" Jul 28 15:22:24.728216369 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:24Z" level=warning msg="[f:8199,l:10735,t:2361,c:10734,a:10734] [00010:51496] t2365 sent RequestVote to n60202" file=" Jul 28 15:22:24.728258509 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:24Z" level=warning msg="[f:8199,l:10735,t:2361,c:10734,a:10734] [00010:51496] t2365 sent RequestVote to n01307" file=" Jul 28 15:22:24.728292720 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:27Z" level=warning msg="[f:8199,l:10735,t:2361,c:10734,a:10734] [00010:51496] t2366 became candidate" file=" Jul 28 15:22:27.481868519 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:27Z" level=warning msg="[f:8199,l:10735,t:2361,c:10734,a:10734] [00010:51496] t2366 received RequestVoteResp from n51496" file=" Jul 28 15:22:27.481940310 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:27Z" level=warning msg="[f:8199,l:10735,t:2361,c:10734,a:10734] [00010:51496] t2366 sent RequestVote to n60202" file=" Jul 28 15:22:27.481984391 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:27Z" level=warning msg="[f:8199,l:10735,t:2361,c:10734,a:10734] [00010:51496] t2366 sent RequestVote to n01307" file=" Jul 28 15:22:27.482018461 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:28Z" level=warning msg="[f:8199,l:10735,t:2361,c:10734,a:10734] [00010:51496] t2366 received RequestVoteResp from n60202" file=" Jul 28 15:22:28.125039535 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:22:28Z" level=warning msg="[f:8199,l:10735,t:2361,c:10734,a:10734] [00010:51496] t2366 received 2 votes and 0 rejections, quorum is 2" file=" Jul 28 15:22:28.125117556 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:24:22Z" level=info msg="Dockie: Sync" file=" Jul 28 15:24:22.066311676 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:27:57Z" level=info msg="Dockie: Sync" file=" Jul 28 15:27:57.616386652 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:31:32Z" level=info msg="Dockie: Sync" file=" Jul 28 15:31:32.717047901 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:35:09Z" level=info msg="Dockie: Sync" file=" Jul 28 15:35:09.440670294 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:38:45Z" level=info msg="Dockie: Sync" file=" Jul 28 15:38:45.457274155 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:42:21Z" level=info msg="Dockie: Sync" file=" Jul 28 15:42:21.661200559 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:45:58Z" level=info msg="Dockie: Sync" file=" Jul 28 15:45:58.835846313 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:49:35Z" level=info msg="Dockie: Sync" file=" Jul 28 15:49:35.272543849 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:51:50Z" level=warning msg="[f:8199,l:10736,t:2366,c:10736,a:10736] [00010:51496] t2366 received RequestVote with higher term (2367) from n60202" file=" Jul 28 15:51:50.798619841 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:51:50Z" level=warning msg="[f:8199,l:10736,t:2366,c:10736,a:10736] [00010:51496] t2366 become followerKE after receiving higher term from n60202" file=" Jul 28 15:51:50.798817963 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:51:50Z" level=warning msg="[f:8199,l:10736,t:2366,c:10736,a:10736] [00010:51496] t2367 cast vote from n60202 index 10736 term 2367, log term: 2366" file=" Jul 28 15:51:50.798890544 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:51:57Z" level=warning msg="[f:8199,l:10737,t:2367,c:10736,a:10736] [00010:51496] t2368 became candidate" file=" Jul 28 15:51:57.568882482 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:51:57Z" level=warning msg="[f:8199,l:10737,t:2367,c:10736,a:10736] [00010:51496] t2368 received RequestVoteResp from n51496" file=" Jul 28 15:51:57.568959823 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:51:57Z" level=warning msg="[f:8199,l:10737,t:2367,c:10736,a:10736] [00010:51496] t2368 sent RequestVote to n60202" file=" Jul 28 15:51:57.569004343 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:51:57Z" level=warning msg="[f:8199,l:10737,t:2367,c:10736,a:10736] [00010:51496] t2368 sent RequestVote to n01307" file=" Jul 28 15:51:57.569041003 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:51:59Z" level=warning msg="[f:8199,l:10737,t:2367,c:10736,a:10736] [00010:51496] t2369 became candidate" file=" Jul 28 15:51:59.941627734 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:51:59Z" level=warning msg="[f:8199,l:10737,t:2367,c:10736,a:10736] [00010:51496] t2369 received RequestVoteResp from n51496" file=" Jul 28 15:51:59.941720555 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:51:59Z" level=warning msg="[f:8199,l:10737,t:2367,c:10736,a:10736] [00010:51496] t2369 sent RequestVote to n60202" file=" Jul 28 15:51:59.941780936 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:51:59Z" level=warning msg="[f:8199,l:10737,t:2367,c:10736,a:10736] [00010:51496] t2369 sent RequestVote to n01307" file=" Jul 28 15:51:59.941836207 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:02Z" level=warning msg="[f:8199,l:10737,t:2367,c:10736,a:10736] [00010:51496] t2370 became candidate" file=" Jul 28 15:52:02.625963535 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:02Z" level=warning msg="[f:8199,l:10737,t:2367,c:10736,a:10736] [00010:51496] t2370 received RequestVoteResp from n51496" file=" Jul 28 15:52:02.626134716 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:02Z" level=warning msg="[f:8199,l:10737,t:2367,c:10736,a:10736] [00010:51496] t2370 sent RequestVote to n60202" file=" Jul 28 15:52:02.626182357 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:02Z" level=warning msg="[f:8199,l:10737,t:2367,c:10736,a:10736] [00010:51496] t2370 sent RequestVote to n01307" file=" Jul 28 15:52:02.626240568 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:06Z" level=warning msg="[f:8199,l:10737,t:2367,c:10736,a:10736] [00010:51496] t2371 became candidate" file=" Jul 28 15:52:06.322192514 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:06Z" level=warning msg="[f:8199,l:10737,t:2367,c:10736,a:10736] [00010:51496] t2371 received RequestVoteResp from n51496" file=" Jul 28 15:52:06.322264215 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:06Z" level=warning msg="[f:8199,l:10737,t:2367,c:10736,a:10736] [00010:51496] t2371 sent RequestVote to n60202" file=" Jul 28 15:52:06.322310656 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:06Z" level=warning msg="[f:8199,l:10737,t:2367,c:10736,a:10736] [00010:51496] t2371 sent RequestVote to n01307" file=" Jul 28 15:52:06.322348366 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:09Z" level=warning msg="[f:8199,l:10737,t:2367,c:10736,a:10736] [00010:51496] t2372 became candidate" file=" Jul 28 15:52:09.549730853 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:09Z" level=warning msg="[f:8199,l:10737,t:2367,c:10736,a:10736] [00010:51496] t2372 received RequestVoteResp from n51496" file=" Jul 28 15:52:09.549856695 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:09Z" level=warning msg="[f:8199,l:10737,t:2367,c:10736,a:10736] [00010:51496] t2372 sent RequestVote to n60202" file=" Jul 28 15:52:09.549899715 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:09Z" level=warning msg="[f:8199,l:10737,t:2367,c:10736,a:10736] [00010:51496] t2372 sent RequestVote to n01307" file=" Jul 28 15:52:09.549955506 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:13Z" level=warning msg="[f:8199,l:10737,t:2367,c:10736,a:10736] [00010:51496] t2373 became candidate" file=" Jul 28 15:52:13.821782026 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:13Z" level=warning msg="[f:8199,l:10737,t:2367,c:10736,a:10736] [00010:51496] t2373 received RequestVoteResp from n51496" file=" Jul 28 15:52:13.821857057 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:13Z" level=warning msg="[f:8199,l:10737,t:2367,c:10736,a:10736] [00010:51496] t2373 sent RequestVote to n01307" file=" Jul 28 15:52:13.821905178 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:13Z" level=warning msg="[f:8199,l:10737,t:2367,c:10736,a:10736] [00010:51496] t2373 sent RequestVote to n60202" file=" Jul 28 15:52:13.821940448 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:14Z" level=warning msg="[f:8199,l:10737,t:2367,c:10736,a:10736] [00010:51496] t2373 received RequestVoteResp from n01307" file=" Jul 28 15:52:14.494703209 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:52:14Z" level=warning msg="[f:8199,l:10737,t:2367,c:10736,a:10736] [00010:51496] t2373 received 2 votes and 0 rejections, quorum is 2" file=" Jul 28 15:52:14.494952252 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:09Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2373 received Heartbeat with higher term (2374) from n60202" file=" Jul 28 15:53:09.600994779 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:09Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2373 become follower after receiving higher term from n60202" file=" Jul 28 15:53:09.601076240 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:13Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2375 became candidate" file=" Jul 28 15:53:13.214629140 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:13Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2375 received RequestVoteResp from n51496" file=" Jul 28 15:53:13.214702421 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:13Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2375 sent RequestVote to n60202" file=" Jul 28 15:53:13.214768912 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:13Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2375 sent RequestVote to n01307" file=" Jul 28 15:53:13.214805652 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:13Z" level=info msg="Dockie: Sync" file=" Jul 28 15:53:13.217486892 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T15:53:16Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2376 became candidate" file=" Jul 28 15:53:16.020163852 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:16Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2376 received RequestVoteResp from n51496" file=" Jul 28 15:53:16.020255003 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:16Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2376 sent RequestVote to n60202" file=" Jul 28 15:53:16.020300984 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:16Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2376 sent RequestVote to n01307" file=" Jul 28 15:53:16.020337884 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:20Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2377 became candidate" file=" Jul 28 15:53:20.051537369 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:20Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2377 received RequestVoteResp from n51496" file=" Jul 28 15:53:20.051618260 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:20Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2377 sent RequestVote to n60202" file=" Jul 28 15:53:20.051666931 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:20Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2377 sent RequestVote to n01307" file=" Jul 28 15:53:20.051709531 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:23Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2378 became candidate" file=" Jul 28 15:53:23.311166610 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:23Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2378 received RequestVoteResp from n51496" file=" Jul 28 15:53:23.311242991 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:23Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2378 sent RequestVote to n60202" file=" Jul 28 15:53:23.311289151 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:23Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2378 sent RequestVote to n01307" file=" Jul 28 15:53:23.311325102 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:26Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2379 became candidate" file=" Jul 28 15:53:26.323530353 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:26Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2379 received RequestVoteResp from n51496" file=" Jul 28 15:53:26.323644074 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:26Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2379 sent RequestVote to n60202" file=" Jul 28 15:53:26.323704965 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:26Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2379 sent RequestVote to n01307" file=" Jul 28 15:53:26.323775226 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:30Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2380 became candidate" file=" Jul 28 15:53:30.189643643 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:30Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2380 received RequestVoteResp from n51496" file=" Jul 28 15:53:30.189717754 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:30Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2380 sent RequestVote to n60202" file=" Jul 28 15:53:30.189775395 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:30Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2380 sent RequestVote to n01307" file=" Jul 28 15:53:30.189812025 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:33Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2381 became candidate" file=" Jul 28 15:53:33.627600127 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:33Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2381 received RequestVoteResp from n51496" file=" Jul 28 15:53:33.627685958 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:33Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2381 sent RequestVote to n60202" file=" Jul 28 15:53:33.627730788 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:33Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2381 sent RequestVote to n01307" file=" Jul 28 15:53:33.627794589 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:37Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2382 became candidate" file=" Jul 28 15:53:37.853189937 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:37Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2382 received RequestVoteResp from n51496" file=" Jul 28 15:53:37.853258427 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:37Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2382 sent RequestVote to n60202" file=" Jul 28 15:53:37.853299908 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:37Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2382 sent RequestVote to n01307" file=" Jul 28 15:53:37.853336488 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:40Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2383 became candidate" file=" Jul 28 15:53:40.442310632 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:40Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2383 received RequestVoteResp from n51496" file=" Jul 28 15:53:40.442385233 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:40Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2383 sent RequestVote to n60202" file=" Jul 28 15:53:40.442430063 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:40Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2383 sent RequestVote to n01307" file=" Jul 28 15:53:40.442466414 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:44Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2384 became candidate" file=" Jul 28 15:53:44.988154379 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:44Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2384 received RequestVoteResp from n51496" file=" Jul 28 15:53:44.988229199 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:44Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2384 sent RequestVote to n60202" file=" Jul 28 15:53:44.988278630 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:44Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2384 sent RequestVote to n01307" file=" Jul 28 15:53:44.988312740 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:48Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2385 became candidate" file=" Jul 28 15:53:48.377135437 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:48Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2385 received RequestVoteResp from n51496" file=" Jul 28 15:53:48.377209818 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:48Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2385 sent RequestVote to n60202" file=" Jul 28 15:53:48.377255008 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:48Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2385 sent RequestVote to n01307" file=" Jul 28 15:53:48.377291609 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:52Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2386 became candidate" file=" Jul 28 15:53:52.212805432 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:52Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2386 received RequestVoteResp from n51496" file=" Jul 28 15:53:52.212879592 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:52Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2386 sent RequestVote to n60202" file=" Jul 28 15:53:52.212926523 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:52Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2386 sent RequestVote to n01307" file=" Jul 28 15:53:52.212961533 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:56Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2387 became candidate" file=" Jul 28 15:53:56.188608621 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:56Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2387 received RequestVoteResp from n51496" file=" Jul 28 15:53:56.188681592 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:56Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2387 sent RequestVote to n60202" file=" Jul 28 15:53:56.188724172 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:56Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2387 sent RequestVote to n01307" file=" Jul 28 15:53:56.188777483 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:59Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2388 became candidate" file=" Jul 28 15:53:59.149289862 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:59Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2388 received RequestVoteResp from n51496" file=" Jul 28 15:53:59.149369823 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:59Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2388 sent RequestVote to n60202" file=" Jul 28 15:53:59.149412004 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:53:59Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2388 sent RequestVote to n01307" file=" Jul 28 15:53:59.149446744 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:03Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2389 became candidate" file=" Jul 28 15:54:03.530829766 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:03Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2389 received RequestVoteResp from n51496" file=" Jul 28 15:54:03.530922197 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:03Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2389 sent RequestVote to n60202" file=" Jul 28 15:54:03.530971137 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:03Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2389 sent RequestVote to n01307" file=" Jul 28 15:54:03.531010368 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:06Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2390 became candidate" file=" Jul 28 15:54:06.945358100 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:06Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2390 received RequestVoteResp from n51496" file=" Jul 28 15:54:06.945430871 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:06Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2390 sent RequestVote to n60202" file=" Jul 28 15:54:06.945475421 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:06Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2390 sent RequestVote to n01307" file=" Jul 28 15:54:06.945511252 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:10Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2391 became candidate" file=" Jul 28 15:54:10.753999231 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:10Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2391 received RequestVoteResp from n51496" file=" Jul 28 15:54:10.754154263 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:10Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2391 sent RequestVote to n01307" file=" Jul 28 15:54:10.754199393 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:10Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2391 sent RequestVote to n60202" file=" Jul 28 15:54:10.754235754 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:15Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2392 became candidate" file=" Jul 28 15:54:15.157056078 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:15Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2392 received RequestVoteResp from n51496" file=" Jul 28 15:54:15.157146909 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:15Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2392 sent RequestVote to n60202" file=" Jul 28 15:54:15.157189810 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:15Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2392 sent RequestVote to n01307" file=" Jul 28 15:54:15.157224530 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:19Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2393 became candidate" file=" Jul 28 15:54:19.286375802 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:19Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2393 received RequestVoteResp from n51496" file=" Jul 28 15:54:19.286449953 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:19Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2393 sent RequestVote to n60202" file=" Jul 28 15:54:19.286495814 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:19Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2393 sent RequestVote to n01307" file=" Jul 28 15:54:19.286532154 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:22Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2394 became candidate" file=" Jul 28 15:54:22.939522785 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:22Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2394 received RequestVoteResp from n51496" file=" Jul 28 15:54:22.939588535 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:22Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2394 sent RequestVote to n60202" file=" Jul 28 15:54:22.939650896 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:22Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2394 sent RequestVote to n01307" file=" Jul 28 15:54:22.939692806 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:28Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2395 became candidate" file=" Jul 28 15:54:28.260348710 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:28Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2395 received RequestVoteResp from n51496" file=" Jul 28 15:54:28.260418001 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:28Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2395 sent RequestVote to n60202" file=" Jul 28 15:54:28.260455052 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:28Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2395 sent RequestVote to n01307" file=" Jul 28 15:54:28.260482492 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:32Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2396 became candidate" file=" Jul 28 15:54:32.160401934 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:32Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2396 received RequestVoteResp from n51496" file=" Jul 28 15:54:32.160478105 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:32Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2396 sent RequestVote to n60202" file=" Jul 28 15:54:32.160542995 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:32Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2396 sent RequestVote to n01307" file=" Jul 28 15:54:32.160581996 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2396 received RequestVoteResp rejection from n01307" file=" Jul 28 15:54:35.823651544 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2396 received 1 votes and 1 rejections, quorum is 2" file=" Jul 28 15:54:35.823737285 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2396 received RequestVoteResp rejection from n60202" file=" Jul 28 15:54:35.960098135 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:35Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2396 received 1 votes and 2 rejections, quorum is 2" file=" Jul 28 15:54:35.960158786 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:37Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2396 received RequestVote with higher term (2397) from n60202" file=" Jul 28 15:54:37.469824742 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:37Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2396 become followerKE after receiving higher term from n60202" file=" Jul 28 15:54:37.469900692 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:54:37Z" level=warning msg="[f:8199,l:10738,t:2373,c:10738,a:10738] [00010:51496] t2397 cast vote from n60202 index 10739 term 2397, log term: 2374" file=" Jul 28 15:54:37.469954963 om/jkassis/[email protected]/entry.go: 313 "
time="2021-07-28T15:56:49Z" level=info msg="Dockie: Sync" file=" Jul 28 15:56:49.682613319 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T16:00:28Z" level=info msg="Dockie: Sync" file=" Jul 28 16:00:28.131953527 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T16:04:02Z" level=info msg="Dockie: Sync" file=" Jul 28 16:04:02.936884708 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T16:07:39Z" level=info msg="Dockie: Sync" file=" Jul 28 16:07:39.491965128 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T16:11:12Z" level=info msg="Dockie: Sync" file=" Jul 28 16:11:12.575630333 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T16:14:50Z" level=info msg="Dockie: Sync" file=" Jul 28 16:14:50.583738853 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T16:18:29Z" level=info msg="Dockie: Sync" file=" Jul 28 16:18:29.168500347 m/jkassis/[email protected]/logger.go: 192 "
time="2021-07-28T16:22:05Z" level=info msg="Dockie: Sync" file=" Jul 28 16:22:05.085356519 m/jkassis/[email protected]/logger.go: 192 "

Jul 29 '21 02:07 jkassis

            - name: RAFT_SERVER_RTT_MILLIS
              value: '10'
            - name: RAFT_SERVER_RTTs_PER_ELECTION
              value: '200'
            - name: RAFT_SERVER_RTTs_PER_HEARTBEAT
              value: '20'

10ms rtt, 200ms heartbeat, 2000ms election

Jul 29 '21 02:07 jkassis

@jkassis thanks for the log above.

If possible, a couple extra inputs I'd like to get from you -

INFO level logs from all involved machines (seems to be 3 replicas in your deployment?) as well
when you felt that your raft cluster got stuck in such election loop, how long did you wait? did it got stuck there forever or just much longer than expected?

thanks!

Jul 29 '21 02:07 lni

thanks as always for quick replies @lni. overall the system has been durable. recovery with no reboots. my subjective experience when I noticed the issue was that it resolved within a minute or two, but i could be wrong. i don't have good external metrics on this availability.

i will flip to info level logging.

Jul 29 '21 02:07 jkassis

good to know it didn't get stuck there forever.

INFO level log will definitely help.

Jul 29 '21 04:07 lni

still working on the info logging, had to instrument a lot of env vars and cli stuff to pass the new log level down. hopefully coming up today. meanwhile i did check the RTT with ping. looks ok...

/opt/app-root $ ping dockie-0.dockie-int.fg.svc.cluster.local
PING dockie-0.dockie-int.fg.svc.cluster.local (10.128.241.223): 56 data bytes
64 bytes from 10.128.241.223: seq=0 ttl=42 time=0.544 ms
64 bytes from 10.128.241.223: seq=1 ttl=42 time=0.562 ms
64 bytes from 10.128.241.223: seq=2 ttl=42 time=0.446 ms
64 bytes from 10.128.241.223: seq=3 ttl=42 time=0.497 ms
64 bytes from 10.128.241.223: seq=4 ttl=42 time=0.571 ms
64 bytes from 10.128.241.223: seq=5 ttl=42 time=0.458 ms
64 bytes from 10.128.241.223: seq=6 ttl=42 time=0.476 ms
64 bytes from 10.128.241.223: seq=7 ttl=42 time=0.427 ms
64 bytes from 10.128.241.223: seq=8 ttl=42 time=0.453 ms
64 bytes from 10.128.241.223: seq=9 ttl=42 time=0.455 ms
64 bytes from 10.128.241.223: seq=10 ttl=42 time=0.547 ms
64 bytes from 10.128.241.223: seq=11 ttl=42 time=0.427 ms
64 bytes from 10.128.241.223: seq=12 ttl=42 time=0.493 ms
64 bytes from 10.128.241.223: seq=13 ttl=42 time=0.548 ms
64 bytes from 10.128.241.223: seq=14 ttl=42 time=0.479 ms
64 bytes from 10.128.241.223: seq=15 ttl=42 time=0.432 ms
64 bytes from 10.128.241.223: seq=16 ttl=42 time=0.466 ms
64 bytes from 10.128.241.223: seq=17 ttl=42 time=0.454 ms
64 bytes from 10.128.241.223: seq=18 ttl=42 time=0.494 ms
64 bytes from 10.128.241.223: seq=19 ttl=42 time=0.444 ms
64 bytes from 10.128.241.223: seq=20 ttl=42 time=0.488 ms
^C
--- dockie-0.dockie-int.fg.svc.cluster.local ping statistics ---
21 packets transmitted, 21 packets received, 0% packet loss
round-trip min/avg/max = 0.427/0.483/0.571 ms
/opt/app-root $ ^C
/opt/app-root $ ^C
/opt/app-root $ ping dockie-1.dockie-int.fg.svc.cluster.local
PING dockie-1.dockie-int.fg.svc.cluster.local (10.130.87.216): 56 data bytes
64 bytes from 10.130.87.216: seq=0 ttl=42 time=0.922 ms
64 bytes from 10.130.87.216: seq=1 ttl=42 time=1.237 ms
64 bytes from 10.130.87.216: seq=2 ttl=42 time=1.951 ms
64 bytes from 10.130.87.216: seq=3 ttl=42 time=0.831 ms
64 bytes from 10.130.87.216: seq=4 ttl=42 time=0.811 ms
64 bytes from 10.130.87.216: seq=5 ttl=42 time=0.861 ms
64 bytes from 10.130.87.216: seq=6 ttl=42 time=0.785 ms
64 bytes from 10.130.87.216: seq=7 ttl=42 time=0.897 ms
64 bytes from 10.130.87.216: seq=8 ttl=42 time=0.842 ms
64 bytes from 10.130.87.216: seq=9 ttl=42 time=0.769 ms

Aug 03 '21 00:08 jkassis

thanks. no rush.

Aug 03 '21 02:08 lni

@lni I am also having to deal with this issue. The cluster still works, but the requesting node won't ever stop until shutdown or actually becoming the leader. It happens very frequently on my (heavily scheduled) test clusters. Several times a day.

I haven't put much effort into looking what's wrong so far. I recently started to make screenshots, when it does happen. I don't think they are much helpful however, so just take this is another +1.

Screenshots: https://imgur.com/a/97fqz05

Dec 21 '21 01:12 intrntbrn

@intrntbrn Thanks for reporting this issue.

From what I can see from those screenshots, it seems to me that you probably don't have the CheckQuorum feature enabled in your config instance used when calling StartCluster/StartConcurrentCluster/StartOnDiskCluster.

Could you please check your code and confirm? Thanks.

Dec 21 '21 15:12 lni

No, CheckQuorum has always been enabled.

Dec 21 '21 16:12 intrntbrn

@intrntbrn in your snapshots, quite often there are 4 nodes involved.

Are you trying to use 3 nodes raft groups and had 4th node added to some raft groups? Did you successfully remove an existing node after adding the 4th node?

Dec 22 '21 14:12 lni

I really don't know what happened exactly in these cases. As i said, i haven't looked into it, because i am currently very busy with implementing my application and for me it seems like it's a bug in dragonboat and has nothing to do with my application. So i've simply ignored it so far.

Some background:

I am implementing a distributed kv store using raft groups. Every raft group/cluster is responsible for a kv range (start, end]. New clusters are created by a split operation (start, split], (split, end].

My cluster is controlled by a global scheduler that tries to balance each node/store. It makes sure that leaderships are equally distributed, adds or replaces replicas, etc...

Usually these are all 3 node raft groups, but the cluster (the very first initial raft group) can be bootstrapped by a single node. So often the very first group is created by a single node. Every non initial node is added as observer first, and will be automatically promoted once it catches up with the leader. There might be cases where the scheduler accidentally added a 4th node. But most likely the scheduler decided to replace a store with another (as more stores got online) to balance replica- or leadercount on each store. In this case the 4th node is added before the "3rd" gets removed. If the to be removed node is the leader, it will request leader transfer before removing the node. I have not yet discovered an issue with removing a node. When it says in the log "remove shard x", then this store was just removed via raft membership change (NodeUnloaded event happened) in cluster with id x, like it does on screenshot 1 in bottom left.

I hope this helps a bit, as i am currently still very busy, but i will try to log every scheduler operation and raft membership change in a few days, so i can provide proper logs of what's actually going on, when this bug occurs.

Dec 22 '21 17:12 intrntbrn

@intrntbrn I think I figured out what happened in your setup.

In this case the 4th node is added before the "3rd" gets removed.

When you have your 3rd node removed, there is no guarantee that the 3rd node being removed will receive that particular membership change raft entry. This is not a bug of the implementation, it is just the way how the protocol works. When this happens, that "3rd" node would become a zombie style node - the node itself still believes that it is a part of its raft group, but other 3 nodes don't include it as a member of the raft group. That zombie 3rd node would keep increasing its term value and keep sending vote request messages to other nodes. Such vote requests are ignored by other nodes, as the leader is known to be available. Other nodes won't respond to the zombie "3rd" node as they don't consider the zombie a part of the raft group (and thus no longer have the routing information required to deliver the message).

Extra mechanism is required to stop the zombies in your application. For example, in my own system and dragonboat's testing framework, I have my monitoring system detecting such zombies and send application level messages to the machines hosting the zombies to get them stopped by invoking NodeHost.StopNode() or NodeHost.StopCluster().

Please note that as described above, other than keep wasting a little bit resources, such zombies are not causing any real harm.

Jan 02 '22 16:01 lni

@lni thank you for looking into the issue. I've applied a workaround over a week ago and i haven't come across the issue again. It's not great having to rely on extra mechanism, but it's fixed, i guess. There should be at least a comment, that NodeUnloaded might not fire on the removed node, so others are also aware. A BecameCandidate event would also help to workaround this issue probably.

Jan 17 '22 00:01 intrntbrn

@intrntbrn

Would you like to send in a PR to improve the docs in this aspect? Sharing your experience with others with be really useful for other users. Will be more than happy to help if you need anything from my side.

Thanks.

Jan 18 '22 02:01 lni