Automate steps for e2e testing

[jdef]

trafficstars

Via slack convo:

smoke tests: (a) start up an etcd cluster; make sure that the number of expected servers is brought online (b) kill a server, make sure that etcd-mesos brings it back online (c) kill a node; make sure that etcd-mesos can cope with that (assuming that normal cluster size is >= 3 servers) (d) bring the node back online; etcd should launch a server on it (e) simulate intermittent network partitions between agent nodes running mesos and etcd server (e.g. by using ip tables to drop packets on mesos and etcd ports) (f) while performing the server/node kills and/or network failures, have a process that continuously increments some counter value in etcd via its HTTP API, and then queries it and compares the result w/ was expected.

The universe package has an example of using an etcd-proxy to front access to the etcd cluster, probably makes testing (f) easier.

[pires]

(e) and (f) seem a fit for some Jepsen integration.

[pires]

Even though it's not Go, here's a project that one may leverage to implement e2e testing, Shakedown.