Support multiple etcd backend endpoints

[alexbrand]

Currently, only one etcd endpoint can be provided to Contiv. Ideally, I should be able to provide multiple etcd endpoints in case one of my etcd nodes in the cluster goes down.

[rchirakk]

contiv should point to etcd running in the localhost/same host and etcd cluster provides high availability. contiv-installer work is in progress to do this.

[alexbrand]

This would work, but it requires the colocation of the contiv pods with the etcd processes on the same machine. This might not always be desirable. For example, you could have a dedicated etcd cluster with nothing else running on the machines.

[rchirakk]

@alexbrand etcd will be scheduled to run on all nodes, some of them will be configured in proxy mode. So you still have etcd on the host but not participating in raft consensus.

[alexbrand]

@rchirakk Got it. That should indeed work, but is there any reason why we can't just pass multiple servers to contiv? The etcd client supports it.

[amccormi]

Etcd in proxy mode has the desired effect of learning all of the nodes in the cluster, and using a different node in case of failure. It's either having the client cycle amongst them, or putting it in the proxy instead. Proxy should be adequate.