Cluster management ideas

[marmar63]

Clustering is configured to use a simple quorum majority - 1/2N+1 - in order to prevent a split brain scenario. While unlikely, it is possible to have two separate sets of even numbers, each trying to gain quorum.

Consider a "dynamic quorum" configuration, where the quorum is calculated by the number of present nodes. If there is an even number of nodes, the voting weight of one node is removed, so as to ensure an odd number of votes, as well as the quorum number (you would need 3 votes instead of 4 at this point, since you have 5 voting nodes). This can be extended so that if a voting node is lost, the weight can be adjusted back to the non-voting node, to ensure an odd number of votes. Additional failures would adjust the votes again, as well as adjust the number needed for quorum. This will ensure maximum uptime of the cluster, down to the point of only two nodes, at which point it becomes a "50/50" situation, where if the voting node goes offline, the cluster goes offline.

Then, any node which fails to gain quorum on its own, will restart its process and attempt to join the existing cluster.