Remove Zookeeper dependency

[jdegoes]

Zookeeper is a large, complex piece of software which requires installation and maintenance. There are many ways for Zookeeper to fail, and Zookeeper requires ongoing management in order to ensure continuous uptime.

While Zookeeper is very appropriate for a large scale distributed storage and analysis system, it is less appropriate for a single node analytics engine like Precog. The dependency just needlessly complicates the architecture, adds more points of failure, results in more lines of code that have to be maintained, and makes Precog more difficult for non-IT wizards to deploy.

In order to simplify the number of moving pieces in Precog, Zookeeper needs to be eliminated as a dependency. Once Kafka is removed (see https://github.com/precog/platform/issues/524), extraction of Zookeeper will be trivial as Zookeeper is used exclusively for tracking offsets.

[nuttycom]

See description of how to produce an environment that is not dependent upon Kafka - Zookeeper is only used for tracking offsets in Kafka for recovery purposes, so removing Kafka obviates the need for Zookeeper entirely.

On Sat, Oct 5, 2013 at 4:13 PM, John A. De Goes [email protected]:

Zookeeper is a large, complex piece of software which requires installation and maintenance. There are many ways for Zookeeper to fail, and Zookeeper requires ongoing management in order to ensure continuous uptime.

While Zookeeper is very appropriate for a large scale distributed storage and analysis system, it is less appropriate for a single node analytics engine like Precog. The dependency just needlessly complicates the architecture, adds more points of failure, results in more lines of code that have to be maintained, and makes Precog more difficult for non-IT wizards to deploy.

In order to simplify the number of moving pieces in Precog, Zookeeper needs to be eliminated as a dependency. Once Kafka is removed (see #524https://github.com/precog/platform/issues/524), extraction of Zookeeper should be a fairly straightforward exercise.

— Reply to this email directly or view it on GitHubhttps://github.com/precog/platform/issues/525 .