radondb-mysql-kubernetes
radondb-mysql-kubernetes copied to clipboard
Published
20 hours ago •
radondb
radondb
Data may inconsistent when followers can't connect to leader.
Describe the problem
Leader should not switch when the leader data is not fully synchronized with the followers.
Replicas 3，v2.1.1
- Continuous read and write data
- 90% of the network packet loss of the simulated Leader node, lasting 60s
Then, the old leader(sample-mysql-0) becomes invalid, The new leader(sample-mysql-1) is selected in the other two nodes，and the data is inconsistent :
/ $ xenoncli cluster gtid
+------------------------------------------+----------+-------+------------------------------------------------+----------------------------------------------+
| ID | Raft | Mysql | Executed_GTID_Set | Retrieved_GTID_Set |
+------------------------------------------+----------+-------+------------------------------------------------+----------------------------------------------+
| sample-mysql-0.sample-mysql.default:8801 | INVALID | ALIVE | 04b00fc5-9061-11ec-9d22-2646bcbce8bb:1-93768, | 04b00fc5-9061-11ec-9d22-2646bcbce8bb:1-93768 |
| | | | b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-107648 | |
+------------------------------------------+----------+-------+------------------------------------------------+----------------------------------------------+
| sample-mysql-2.sample-mysql.default:8801 | FOLLOWER | ALIVE | 04b00fc5-9061-11ec-9d22-2646bcbce8bb:1-93768, | 04b00fc5-9061-11ec-9d22-2646bcbce8bb:1-93768 |
| | | | b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-106624 | |
+------------------------------------------+----------+-------+------------------------------------------------+----------------------------------------------+
| sample-mysql-1.sample-mysql.default:8801 | LEADER | ALIVE | 04b00fc5-9061-11ec-9d22-2646bcbce8bb:1-93768, | |
| | | | b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-106624 | |
+------------------------------------------+----------+-------+------------------------------------------------+----------------------------------------------+
xenon log of old leader(sample-mysql-0):
2022/02/18 03:06:40.060491 trace.go:37: [ERROR] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:2].send.heartbeat.to.peer[sample-mysql-2.sample-mysql.default:8801].new.client.error[dial tcp: i/o timeout]
2022/02/18 03:06:40.060554 trace.go:32: [WARNING] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:2].heartbeat.acks.granted[1].less.than.quorums[2].lessHtAcks[1].maxLessHtAcks[5]
2022/02/18 03:06:40.060568 trace.go:37: [ERROR] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:2].send.heartbeat.to.peer[sample-mysql-1.sample-mysql.default:8801].new.client.error[dial tcp: i/o timeout]
2022/02/18 03:06:40.361432 xenon.go:34: [ERROR] api.v1.xenon.ping.error:get.client.error[dial tcp: i/o timeout]
2022/02/18 03:06:40 http: panic serving 10.10.242.188:52766: runtime error: invalid memory address or nil pointer dereference
goroutine 1487 [running]:
net/http.(*conn).serve.func1(0xc0002161e0)
/usr/local/go/src/net/http/server.go:1804 +0x153
panic(0x87e1a0, 0xbefaf0)
/usr/local/go/src/runtime/panic.go:971 +0x499
ctl/v1.xenonPingHandler(0xc0001142f0, 0xc0001627e0, 0x993a80, 0xc0000a9f08, 0xc0000a9ef0)
/go/src/github.com/radondb/xenon/src/ctl/v1/xenon.go:37 +0x11c
ctl/v1.XenonPingHandler.func1(0x993a80, 0xc0000a9f08, 0xc0000a9ef0)
/go/src/github.com/radondb/xenon/src/ctl/v1/xenon.go:25 +0x51
vendor/github.com/ant0ine/go-json-rest/rest.(*router).AppFunc.func1(0x993a80, 0xc0000a9f08, 0xc0000a9ef0)
/go/src/github.com/radondb/xenon/src/vendor/github.com/ant0ine/go-json-rest/rest/router.go:56 +0x99
vendor/github.com/ant0ine/go-json-rest/rest.(*AuthBasicMiddleware).MiddlewareFunc.func2(0x993a80, 0xc0000a9f08, 0xc0000a9ef0)
/go/src/github.com/radondb/xenon/src/vendor/github.com/ant0ine/go-json-rest/rest/auth_basic.go:73 +0x1cf
vendor/github.com/ant0ine/go-json-rest/rest.adapterFunc.func1(0x992c88, 0xc0005348c0, 0xc0001ce200)
/go/src/github.com/radondb/xenon/src/vendor/github.com/ant0ine/go-json-rest/rest/middleware.go:70 +0xd3
net/http.HandlerFunc.ServeHTTP(0xc000058460, 0x992c88, 0xc0005348c0, 0xc0001ce200)
/usr/local/go/src/net/http/server.go:2049 +0x44
net/http.serverHandler.ServeHTTP(0xc0000760e0, 0x992c88, 0xc0005348c0, 0xc0001ce200)
/usr/local/go/src/net/http/server.go:2867 +0xa3
net/http.(*conn).serve(0xc0002161e0, 0x9937e0, 0xc00020c700)
/usr/local/go/src/net/http/server.go:1932 +0x8cd
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2993 +0x39b
2022/02/18 03:06:42.062034 trace.go:37: [ERROR] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:2].send.heartbeat.to.peer[sample-mysql-1.sample-mysql.default:8801].new.client.error[dial tcp: i/o timeout]
2022/02/18 03:06:42.062048 trace.go:37: [ERROR] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:2].send.heartbeat.to.peer[sample-mysql-2.sample-mysql.default:8801].new.client.error[dial tcp: i/o timeout]
2022/02/18 03:06:42.062186 trace.go:32: [WARNING] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:2].heartbeat.acks.granted[1].less.than.quorums[2].lessHtAcks[2].maxLessHtAcks[5]
2022/02/18 03:06:44.062574 trace.go:37: [ERROR] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:2].send.heartbeat.to.peer[sample-mysql-1.sample-mysql.default:8801].new.client.error[dial tcp: i/o timeout]
2022/02/18 03:06:44.062622 trace.go:32: [WARNING] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:2].heartbeat.acks.granted[1].less.than.quorums[2].lessHtAcks[3].maxLessHtAcks[5]
2022/02/18 03:06:44.062647 trace.go:37: [ERROR] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:2].send.heartbeat.to.peer[sample-mysql-2.sample-mysql.default:8801].new.client.error[dial tcp: i/o timeout]
2022/02/18 03:06:44.386478 api.go:285: [INFO] mysql.slave.status:&{ 0 false false }
2022/02/18 03:06:44.386712 api.go:295: [INFO] mysql.master.status:&{mysql-bin.000005 1072490484 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-106624 0 }
2022/02/18 03:06:46.063395 trace.go:37: [ERROR] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:2].send.heartbeat.to.peer[sample-mysql-2.sample-mysql.default:8801].new.client.error[dial tcp: i/o timeout]
2022/02/18 03:06:46.063402 trace.go:32: [WARNING] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:2].heartbeat.acks.granted[1].less.than.quorums[2].lessHtAcks[4].maxLessHtAcks[5]
2022/02/18 03:06:46.063453 trace.go:37: [ERROR] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:2].send.heartbeat.to.peer[sample-mysql-1.sample-mysql.default:8801].new.client.error[dial tcp: i/o timeout]
2022/02/18 03:06:48.064198 trace.go:37: [ERROR] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:2].send.heartbeat.to.peer[sample-mysql-1.sample-mysql.default:8801].new.client.error[dial tcp: i/o timeout]
2022/02/18 03:06:48.064213 trace.go:37: [ERROR] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:2].send.heartbeat.to.peer[sample-mysql-2.sample-mysql.default:8801].new.client.error[dial tcp: i/o timeout]
2022/02/18 03:06:48.064227 trace.go:32: [WARNING] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:2].heartbeat.acks.granted[1].less.than.quorums[2].lessHtAcks[5].maxLessHtAcks[5]
2022/02/18 03:06:48.064254 trace.go:32: [WARNING] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:2].degrade.to.follower.lessHtAcks[5]>=maxLessHtAcks[5]
2022/02/18 03:06:48.064259 trace.go:32: [WARNING] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:2].degrade.to.follower.stop.the.vip...
2022/02/18 03:06:49.386049 api.go:285: [INFO] mysql.slave.status:&{ 0 false false }
2022/02/18 03:06:49.386328 api.go:295: [INFO] mysql.master.status:&{mysql-bin.000005 1072490484 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-106624 0 }
2022/02/18 03:06:54.386357 api.go:285: [INFO] mysql.slave.status:&{ 0 false false }
2022/02/18 03:06:54.386624 api.go:295: [INFO] mysql.master.status:&{mysql-bin.000005 1072490484 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-106624 0 }
2022/02/18 03:06:59.385731 api.go:285: [INFO] mysql.slave.status:&{ 0 false false }
2022/02/18 03:06:59.385918 api.go:295: [INFO] mysql.master.status:&{mysql-bin.000005 1072490484 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-106624 0 }
2022/02/18 03:07:04.385802 api.go:285: [INFO] mysql.slave.status:&{ 0 false false }
2022/02/18 03:07:04.386025 api.go:295: [INFO] mysql.master.status:&{mysql-bin.000005 1072490484 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-106624 0 }
2022/02/18 03:07:09.386531 api.go:285: [INFO] mysql.slave.status:&{ 0 false false }
2022/02/18 03:07:09.386740 api.go:295: [INFO] mysql.master.status:&{mysql-bin.000005 1072490484 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-106624 0 }
2022/02/18 03:07:14.386306 api.go:285: [INFO] mysql.slave.status:&{ 0 false false }
2022/02/18 03:07:14.386538 api.go:295: [INFO] mysql.master.status:&{mysql-bin.000005 1072490484 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-106624 0 }
2022/02/18 03:07:19.385595 api.go:285: [INFO] mysql.slave.status:&{ 0 false false }
2022/02/18 03:07:19.385783 api.go:295: [INFO] mysql.master.status:&{mysql-bin.000005 1072490484 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-106624 0 }
2022/02/18 03:07:24.386190 api.go:285: [INFO] mysql.slave.status:&{ 0 false false }
2022/02/18 03:07:24.386487 api.go:295: [INFO] mysql.master.status:&{mysql-bin.000005 1072490484 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-106624 0 }
2022/02/18 03:07:29.386144 api.go:285: [INFO] mysql.slave.status:&{ 0 false false }
2022/02/18 03:07:29.386355 api.go:295: [INFO] mysql.master.status:&{mysql-bin.000005 1072490484 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-106624 0 }
2022/02/18 03:07:34.386493 api.go:285: [INFO] mysql.slave.status:&{ 0 false false }
2022/02/18 03:07:34.389835 api.go:295: [INFO] mysql.master.status:&{mysql-bin.000005 1072490484 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-106624 0 }
2022/02/18 03:07:39.367732 cluster.go:98: [WARNING] api.v1.cluster.prepare.to.remove.nodes[sample-mysql-2.sample-mysql.default:8801].from.leader[sample-mysql-0.sample-mysql.default:8801]
2022/02/18 03:07:39.369044 rpc_node.go:60: [WARNING] server.rpc.node.remove:&{From: Nodes:[sample-mysql-2.sample-mysql.default:8801]}
2022/02/18 03:07:39.369232 trace.go:27: [INFO] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:3].check.peers.json.file[/var/lib/xenon/peers.json].stat[name:peers.json, mode:-rwxr-xr-x, size:114, lastmodified:2022-02-18 03:07:39.364133404 +0000 UTC]
2022/02/18 03:07:39.369252 trace.go:32: [WARNING] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:3].removed.peer[sample-mysql-2.sample-mysql.default:8801].from.peers[map[sample-mysql-1.sample-mysql.default:8801:0xc0002030b0]]
2022/02/18 03:07:39.369498 cluster.go:113: [WARNING] api.v1.cluster.remove.nodes.from.leader[sample-mysql-0.sample-mysql.default:8801].done
2022/02/18 03:07:39.386747 api.go:285: [INFO] mysql.slave.status:&{ 0 false false }
2022/02/18 03:07:39.386951 api.go:295: [INFO] mysql.master.status:&{mysql-bin.000005 1072490484 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-106624 0 }
2022/02/18 03:07:39.441159 rpc_node.go:60: [WARNING] server.rpc.node.remove:&{From: Nodes:[sample-mysql-2.sample-mysql.default:8801]}
2022/02/18 03:07:39.454312 trace.go:32: [WARNING] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:3].peer[sample-mysql-2.sample-mysql.default:8801].not.exists.in.peers[map[sample-mysql-1.sample-mysql.default:8801:0xc0002030b0]]
2022/02/18 03:07:39.516594 cluster.go:98: [WARNING] api.v1.cluster.prepare.to.remove.nodes[sample-mysql-2.sample-mysql.default:8801].from.leader[sample-mysql-0.sample-mysql.default:8801]
2022/02/18 03:07:39.517975 rpc_node.go:60: [WARNING] server.rpc.node.remove:&{From: Nodes:[sample-mysql-2.sample-mysql.default:8801]}
2022/02/18 03:07:39.517997 trace.go:32: [WARNING] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:3].peer[sample-mysql-2.sample-mysql.default:8801].not.exists.in.peers[map[sample-mysql-1.sample-mysql.default:8801:0xc0002030b0]]
2022/02/18 03:07:39.518173 cluster.go:113: [WARNING] api.v1.cluster.remove.nodes.from.leader[sample-mysql-0.sample-mysql.default:8801].done
2022/02/18 03:07:39.586465 rpc_node.go:60: [WARNING] server.rpc.node.remove:&{From: Nodes:[sample-mysql-2.sample-mysql.default:8801]}
2022/02/18 03:07:39.586497 trace.go:32: [WARNING] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:3].peer[sample-mysql-2.sample-mysql.default:8801].not.exists.in.peers[map[sample-mysql-1.sample-mysql.default:8801:0xc0002030b0]]
2022/02/18 03:07:39.627511 cluster.go:98: [WARNING] api.v1.cluster.prepare.to.remove.nodes[sample-mysql-2.sample-mysql.default:8801].from.leader[sample-mysql-0.sample-mysql.default:8801]
2022/02/18 03:07:39.628907 rpc_node.go:60: [WARNING] server.rpc.node.remove:&{From: Nodes:[sample-mysql-2.sample-mysql.default:8801]}
2022/02/18 03:07:39.628930 trace.go:32: [WARNING] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:3].peer[sample-mysql-2.sample-mysql.default:8801].not.exists.in.peers[map[sample-mysql-1.sample-mysql.default:8801:0xc0002030b0]]
2022/02/18 03:07:39.629095 cluster.go:113: [WARNING] api.v1.cluster.remove.nodes.from.leader[sample-mysql-0.sample-mysql.default:8801].done
2022/02/18 03:07:39.694519 rpc_node.go:60: [WARNING] server.rpc.node.remove:&{From: Nodes:[sample-mysql-2.sample-mysql.default:8801]}
2022/02/18 03:07:39.694558 trace.go:32: [WARNING] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:3].peer[sample-mysql-2.sample-mysql.default:8801].not.exists.in.peers[map[sample-mysql-1.sample-mysql.default:8801:0xc0002030b0]]
2022/02/18 03:07:39.824667 cluster.go:98: [WARNING] api.v1.cluster.prepare.to.remove.nodes[sample-mysql-2.sample-mysql.default:8801].from.leader[sample-mysql-0.sample-mysql.default:8801]
2022/02/18 03:07:39.826771 rpc_node.go:60: [WARNING] server.rpc.node.remove:&{From: Nodes:[sample-mysql-2.sample-mysql.default:8801]}
2022/02/18 03:07:39.826809 trace.go:32: [WARNING] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:3].peer[sample-mysql-2.sample-mysql.default:8801].not.exists.in.peers[map[sample-mysql-1.sample-mysql.default:8801:0xc0002030b0]]
2022/02/18 03:07:39.827064 cluster.go:113: [WARNING] api.v1.cluster.remove.nodes.from.leader[sample-mysql-0.sample-mysql.default:8801].done
2022/02/18 03:07:39.843363 rpc_node.go:60: [WARNING] server.rpc.node.remove:&{From: Nodes:[sample-mysql-2.sample-mysql.default:8801]}
2022/02/18 03:07:39.843405 trace.go:32: [WARNING] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:3].peer[sample-mysql-2.sample-mysql.default:8801].not.exists.in.peers[map[sample-mysql-1.sample-mysql.default:8801:0xc0002030b0]]
2022/02/18 03:07:44.359816 cluster.go:98: [WARNING] api.v1.cluster.prepare.to.remove.nodes[sample-mysql-2.sample-mysql.default:8801].from.leader[sample-mysql-0.sample-mysql.default:8801]
2022/02/18 03:07:44.361278 rpc_node.go:60: [WARNING] server.rpc.node.remove:&{From: Nodes:[sample-mysql-2.sample-mysql.default:8801]}
2022/02/18 03:07:44.361309 trace.go:32: [WARNING] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:3].peer[sample-mysql-2.sample-mysql.default:8801].not.exists.in.peers[map[sample-mysql-1.sample-mysql.default:8801:0xc0002030b0]]
2022/02/18 03:07:44.361573 cluster.go:113: [WARNING] api.v1.cluster.remove.nodes.from.leader[sample-mysql-0.sample-mysql.default:8801].done
2022/02/18 03:07:44.385577 api.go:285: [INFO] mysql.slave.status:&{ 0 false false }
2022/02/18 03:07:44.385838 api.go:295: [INFO] mysql.master.status:&{mysql-bin.000005 1072490484 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-106624 0 }
2022/02/18 03:07:44.397306 rpc_node.go:60: [WARNING] server.rpc.node.remove:&{From: Nodes:[sample-mysql-2.sample-mysql.default:8801]}
2022/02/18 03:07:44.397343 trace.go:32: [WARNING] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:3].peer[sample-mysql-2.sample-mysql.default:8801].not.exists.in.peers[map[sample-mysql-1.sample-mysql.default:8801:0xc0002030b0]]
2022/02/18 03:07:49.386328 api.go:285: [INFO] mysql.slave.status:&{ 0 false false }
2022/02/18 03:07:49.386485 api.go:295: [INFO] mysql.master.status:&{mysql-bin.000005 1072491705 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-106624 0 }
2022/02/18 03:07:49.416111 cluster.go:53: [WARNING] api.v1.cluster.prepare.to.add.nodes[sample-mysql-2.sample-mysql.default:8801].to.leader[sample-mysql-0.sample-mysql.default:8801]
2022/02/18 03:07:49.417466 rpc_node.go:28: [WARNING] server.rpc.node.add:&{From: Nodes:[sample-mysql-2.sample-mysql.default:8801]}
2022/02/18 03:07:49.417663 trace.go:27: [INFO] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].check.peers.json.file[/var/lib/xenon/peers.json].stat[name:peers.json, mode:-rwxr-xr-x, size:157, lastmodified:2022-02-18 03:07:49.411951679 +0000 UTC]
2022/02/18 03:07:49.417685 trace.go:32: [WARNING] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].add.peer[sample-mysql-2.sample-mysql.default:8801].to.peers[map[sample-mysql-1.sample-mysql.default:8801:0xc0002030b0 sample-mysql-2.sample-mysql.default:8801:0xc00051f020]]
2022/02/18 03:07:49.418417 cluster.go:68: [WARNING] api.v1.cluster.add.nodes.to.leader[sample-mysql-0.sample-mysql.default:8801].done
2022/02/18 03:07:53.594398 trace.go:32: [WARNING] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].leaderStopShellCommand[[-c /scripts/leader-stop.sh]].done
2022/02/18 03:07:53.594432 trace.go:27: [INFO] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].check.semi-sync.thread.stop...
2022/02/18 03:07:53.594438 trace.go:27: [INFO] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].check.gtid.thread.stop...
2022/02/18 03:07:53.594447 trace.go:32: [WARNING] FOLLOWER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].leader.state.machine.exit.done
2022/02/18 03:07:53.594452 trace.go:27: [INFO] FOLLOWER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].start.CheckBrainSplit
2022/02/18 03:07:53.594506 trace.go:32: [WARNING] FOLLOWER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].state.init
2022/02/18 03:07:53.671424 trace.go:32: [WARNING] FOLLOWER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].leaderStopShellCommand[[-c /scripts/leader-stop.sh]].done
2022/02/18 03:07:53.671463 trace.go:32: [WARNING] FOLLOWER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].mysql.waitMysqlDoneAsync.prepare
2022/02/18 03:07:53.671480 trace.go:32: [WARNING] FOLLOWER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].state.machine.run
2022/02/18 03:07:58.594911 trace.go:32: [WARNING] FOLLOWER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].ping.responses[1].is.less.than.half.maybe.brain.split
2022/02/18 03:07:58.598309 trace.go:32: [WARNING] FOLLOWER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].ping.responses[2].is.greater.than.half.again
2022/02/18 03:07:58.912879 trace.go:32: [WARNING] FOLLOWER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].mysql.SetReadOnly.done
2022/02/18 03:07:58.954602 api.go:385: [WARNING] mysql[localhost:3306].SetSlaveGlobalSysVar[tokudb_fsync_log_period=1000;sync_binlog=1000;innodb_flush_log_at_trx_commit=1]
2022/02/18 03:07:58.954637 trace.go:32: [WARNING] FOLLOWER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].mysql.SetSlaveGlobalSysVar.done
2022/02/18 03:07:58.954644 trace.go:32: [WARNING] FOLLOWER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].prepareAsync.done
2022/02/18 03:07:58.960826 trace.go:37: [ERROR] FOLLOWER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].mysql.StartSlave.error[Error 1200: The server is not configured as slave; fix in config file or with CHANGE MASTER TO]
2022/02/18 03:07:58.960930 api.go:285: [INFO] mysql.slave.status:&{ 0 false false }
2022/02/18 03:07:58.961121 api.go:285: [INFO] mysql.slave.status:&{ 0 false false }
2022/02/18 03:07:58.961176 api.go:295: [INFO] mysql.master.status:&{mysql-bin.000005 1073642996 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-107648 0 }
2022/02/18 03:07:58.961335 trace.go:32: [WARNING] FOLLOWER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].init.my.gtid.is:{mysql-bin.000005 1073642996 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-107648 0 }
2022/02/18 03:07:58.961367 api.go:295: [INFO] mysql.master.status:&{mysql-bin.000005 1073642996 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-107648 0 }
2022/02/18 03:07:58.961407 trace.go:32: [WARNING] FOLLOWER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].get.heartbeat.my.gtid.is:{mysql-bin.000005 1073642996 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-107648 0 }
2022/02/18 03:07:58.961770 mysql.go:137: [INFO] mysql.get.local.uuid:[b2c3c5d9-9060-11ec-9817-12c02189a4ce]
2022/02/18 03:07:58.962428 mysql.go:137: [INFO] mysql.get.local.uuid:[b2c3c5d9-9060-11ec-9817-12c02189a4ce]
2022/02/18 03:07:58.962639 api.go:173: [WARNING] follower.gtid[&{mysql-bin.000005 1073642996 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-107648 0 }].bigger.than.remote[&{mysql-bin.000003 1070145394 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-106624 0 }]
2022/02/18 03:07:58.962731 trace.go:32: [WARNING] INVALID[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].get.heartbeat.from[N:sample-mysql-1.sample-mysql.default:8801, V:46, E:4].change.mysql.master
2022/02/18 03:07:58.995761 trace.go:32: [WARNING] INVALID[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].get.heartbeat.change.to.the.new.master[sample-mysql-1.sample-mysql.default:8801].successed
2022/02/18 03:07:58.995790 trace.go:32: [WARNING] INVALID[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].get.heartbeat.from[N:sample-mysql-1.sample-mysql.default:8801, V:46, E:4].update.view
2022/02/18 03:07:58.995797 trace.go:32: [WARNING] INVALID[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].do.updateViewID[FROM:41 TO:46]
2022/02/18 03:07:58.995826 trace.go:32: [WARNING] INVALID[ID:sample-mysql-0.sample-mysql.default:8801, V:46, E:4].follower.state.machine.exit
2022/02/18 03:07:59.073417 trace.go:32: [WARNING] INVALID[ID:sample-mysql-0.sample-mysql.default:8801, V:46, E:4].leaderStopShellCommand[[-c /scripts/leader-stop.sh]].done
2022/02/18 03:07:59.074417 api.go:385: [WARNING] mysql[localhost:3306].SetSlaveGlobalSysVar[tokudb_fsync_log_period=1000;sync_binlog=1000;innodb_flush_log_at_trx_commit=1]
To Reproduce
Expected behavior
Environment:
- RadonDB MySQL version:
http://my-replication-life.blogspot.com/2013/09/loss-less-semi-synchronous-replication.html