ringman icon indicating copy to clipboard operation
ringman copied to clipboard

Examples

Open kotyara85 opened this issue 2 years ago • 1 comments

Is there any examples on how to join nodes into a cluster? Also, I guess it's up to a developer how to rebalance nodes if added / removed? Thanks

kotyara85 avatar Oct 15 '21 05:10 kotyara85

@kotyara85 Thanks for trying this out! The idea here is to establish a cluster of nodes using either barebones Memberlist or Sidecar (which uses Memberlist under the hood) and designate a subset of the nodes as seeds for the rest of the nodes you want to add to the cluster. As long as some seed nodes are up and running, the new nodes will bootstrap from those. For small clusters (say under 100 nodes), all the nodes can be used as seeds. However, for larger clusters, this could make startup too slow because of this issue. As you pointed out, if a subset of the nodes are designated as seeds, then some custom logic is needed to ensure there will always be at least a few seeds that are operational and valid. Otherwise, a split brain situation can occur. When I was working on this, we didn't have a large enough cluster to worry about having too many seed nodes, but I can think of using some sort of pre-deployment hook which checks if the seed list is becoming stale before kicking off the deploy. If it detects a stale seed list, then it can first trigger a seed list refresh across the cluster. Maybe there are better ways of doing this, but not sure. I'd have to check how other systems which rely on Memberlist do this, such as Serf. The Memberlist papers are worth reading as well as some of the technical articles from Hashicorp: https://www.hashicorp.com/blog/making-gossip-more-robust-with-lifeguard, https://www.serf.io/docs/internals/gossip.html etc.

mihaitodor avatar Oct 15 '21 13:10 mihaitodor