Message Queue based on ZooKeeper

NOTICE: This is work in progress and it's not production ready (more like a proof of concept)

NOTICE: It should be easy to rewrite the code in Java by following the same concepts.

Objectives

Build a highly available & reliable message queue. The main focus is not on perfomance!

This queue should be able to survive hardware and software upgrades and failures with no downtime for years :)

The queue should be accesible using both the default zookeeper binary protocol and the zookeeper rest interface.

ZooKeeper Data Tree Structure

<chroot-prefix OR />/ /queue /items/ /item-000-1 /item-050-2 ... /partial - items reserved but not marked as done (consumer failure) /item-1 /item-2 ... /consumers /consumer-1 /active - ephemeral znode /item /consumer-2 /active - ephemeral /item

ZooKeeper Availability

A ZooKeeper cluster with N servers will still be available if CEIL(N/2)-1 nodes fail!

E.g. if you run a cluster with five nodes you can take one offline for maintenance anytime and you will still be able to handle an unexpected failure.

Requirements

Tested with Python 2.6 and zkpython 3.4.0--1, built on 02/19/2011 14:52 GMT

You also need simplejson for the rest interface

How to install zkpython?

Fetch the latest sources from the trunk and run:

$ apt-get install sun-java6-jdk libtool libcppunit-dev automake build-essential ant

$ cd zookeeper $ ant compile compile_jute $ cd src/c/ $ autoreconf -if $ ./configure $ make && sudo make install $ cd ../contrib/zkpython $ ant compile $ sudo ant install

After all this you should be able to do:

$ python -c "import zookeeper;print zookeeper.version" 3.4.0

Fault Injection Testing Scenarios

I'm not going to test the producer and the consumer code. I'm trying to to test the availability and reliability of the ZooKeeper quorum when used for storing the queue data.

Start a ZooKeeper quorum and the fault injector:

$ cd test_5_trunk $ ./clean.sh $ ./start.sh $ python fail.py leader

Push 500 elements:

$ ./put_all.py

or

$ PYTHONPATH=lib.linux-x86_64-2.6/ LD_LIBRARY_PATH=lib.linux-x86_64-2.6/ ./put_all.py

Check 500 elements (also checks that the sequence is strictly increasing):

$ ./check_all.py

or

$ PYTHONPATH=lib.linux-x86_64-2.6/ LD_LIBRARY_PATH=lib.linux-x86_64-2.6/ ./check_all.py

It should work like a charm! :) Keep in mind that you are not going to experience this amount of critical failures in production.

License

Apache License Version 2.0

Resources

Inspired by this article wrote by Henry Robinson on the Cloudera blog:

http://www.cloudera.com/blog/2009/05/building-a-distributed-concurrent-queue-with-apache-zookeeper/