tarantool icon indicating copy to clipboard operation
tarantool copied to clipboard

box.ctl.demote() does not guarantee demoting

Open askalt opened this issue 2 years ago • 5 comments

box.ctl.demote() does not guarantee that the node will not become a leader in the new term.

For example, we have the cluster with 3 nodes and election failover.

$ tt status app
INSTANCE           STATUS      PID
app:election-1     RUNNING     3867795
app:election-2     RUNNING     3867796
app:election-3     RUNNING     3868129

$ echo "box.info.election" | tt connect app:election-2
---
- leader_idle: 0
  leader_name: election-2
  state: leader
  vote: 2
  term: 80
  leader: 2
...

$ echo "box.ctl.demote()" | tt connect app:election-2
---
...

$ echo "box.info.election" | tt connect app:election-2
---
- leader_idle: 0
  leader_name: election-2
  state: leader
  vote: 2
  term: 81
  leader: 2
...

askalt avatar Mar 21 '24 17:03 askalt

Its not a bug, its expected behavior , demote just bump new term when raft enabled.

if you need to prevent a node from becoming a leader in the new round of elections, you need set election_mode to voter.

R-omk avatar Mar 21 '24 21:03 R-omk

I just tried it and had to retry box.ctl.demote() about 10 times before some other leader than the current one got elected. It seems box.ctl.demote() isn't useful in its current form, we should rework it probably.

I suppose the problem with current implementation is that the old leader sees that no-one has requested a vote in the just-persisted term (because it has just written it without voting, and noone had time to see the new term yet), and hence votes for itself, becoming the most probable winner of the election

sergepetrenko avatar Mar 22 '24 08:03 sergepetrenko

@sergepetrenko

and hence votes for itself

it can prohibit voting for itself for some time, for example election_timeout x2(x4) (it's like temporarily turning on the "voter" mode )

R-omk avatar Mar 22 '24 08:03 R-omk

it can prohibit voting for itself for some time, for example election_timeout x2(x4) (it's like temporarily turning on the "voter" mode )

@R-omk Yep, I thought about the same solution. Another option is to prohibit voting for self for the entire term (but still allow to start new term after an election_timeout)

sergepetrenko avatar Mar 22 '24 08:03 sergepetrenko

Another option is to prohibit voting for self for the entire term

👍

Gerold103 avatar Mar 22 '24 18:03 Gerold103