NuRaft with kubernetes

[kishorekrd]

Hi, I am creating NuRaft cluster using Kubernetes. Each container has local IP address (will change after reboot) and service IP address(persistent across reboots). I initialized NuRaft cluster suing Service IP addresses, but the RPC request comes from the local IP address from peer Nodes. Is there any problem with this configuration? Can I configure a node with multiple IP addresses? Are there any dos and don'ts with kubernetes?

I see that each Node getting connection request from other peer node local IP address and getting established, but after this it is trying to connect to the same peer node with its Service IP address and getting connection errors. This causing flapping and leading to election time outs. Am I missing anything?

[greensky00]

eBay internally uses NuRaft + Kubernetes and there is no such problem, each container always sees the correct IP address. I guess it is related to your environment settings.

And even though an RPC server sees a local IP address from an incoming request, it should not affect the connection to the peer, as the connection is established based on the endpoint of srv_confg. As long as srv_config has the correct endpoint (i.e., service IP), NuRaft always uses that endpoint regardless of incoming IP address.

[kishorekrd]

Thanks for the response. What are the ideal values that you recommend for heart_beat_interval_, election_timeout_lower_bound_, election_timeout_upper_bound_ parameters while running on Kubernetes on a same network?

[greensky00]

We use around 1 second, 2 seconds, and 4 seconds for heartbeat, lower and upper bounds.

[kishorekrd]

is that means, for 2 heart beat misses, it will trigger the election? What is the reason for 1 second heart_beat_interval_? isn't it too big?

[greensky00]

It should be determined considering two factors:

1. How often will the leader node actually die?
2. How often will there be a short network hiccup?

Usually 2) is much more frequent than 1). If we set the above Raft intervals shorter than the network hiccup duration, that will cause lots of unnecessary leader changes even though the previous leader has no problem, which is not a preferred situation as the system becomes unstable.

Hence, the intervals need to be balanced between two factors, and the above numbers are our empirical optimum.