moxi - a memcached/membase proxy with energy and pep

Dependencies:

-- libevent, http://www.monkey.org/~provos/libevent/ (libevent-dev)

-- libconflate, http://github.com/northscale/libconflate

-- libvbucket, http://github.com/northscale/libvbucket

As a backwards-compatible (ketama/consistent-hashing) alternative to libvbucket, you may instead use libmemcached instead of libvbucket...

-- libmemcached, http://tangent.org/552/libmemcached.html

To use moxi against membase, however, you'll want libvbucket.

To compile moxi (assuming you want libvbucket), after you got the dependencies built and installed:

./config/autorun.sh ./configure make

For example, if libevent is installed in /opt/local, you'd do...

./config/autorun.sh ./configure --with-libevent=/opt/local make

Using moxi:

To have moxi load a "vBucket" json configuration from a REST/HTTP server, try...

moxi http://host:port/url/of/vBucketServerMapJSON

Against a NorthScale server, for example, this would look like...

moxi http://host:8080/pools/default/bucketsStreamingConfig/default

The above, also, is just shorthand (assuming you have no other flags specified), for the following explicit command-line...

moxi -z url=http://host:8080/url/of/vBucketServerMap.json

To get more command line usage info:

moxi -h

File-based configuration:

You may also provide a configuration file to moxi that holds a static vBucket server map. The file (such as vbucket1.cfg) would look like...

11211 = { "hashAlgorithm": "CRC", "numReplicas": 0, "serverList": ["memcached_svr1:11311"], "vBucketMap": [ [0], [0] ] }

The above configuration would tell moxi to listen on port 11211, and proxy to memcached_svr1:11311. To use a static configuration file, you would start moxi like...

moxi -z ./vbucket1.cfg

The "./" path prefix is required so that moxi knows you're passing in a config file.

Tests:

To test that moxi still behaves like memcached

and passes all the tests that memcached passes...

make test

To test moxi in a simple proxy topology of...

client <-> moxi <-> memcached

./t/moxi.pl

To test moxi in a chained proxy topology of...

client <-> moxi <-> moxi <-> moxi <-> memcached

./t/moxi.pl chain

To test moxi in a fanout topology of...

client <---> moxi <---> memcached

|-> memcached

|-> memcached

|-> memcached

./t/moxi.pl fanout

To test moxi in a fanout and back in again topology of...

client <---> moxi <---> moxi <-> memcached

|-> moxi <-|

|-> moxi <-|

|-> moxi <-|

./t/moxi.pl fanoutin

To test moxi proxy cases...

./moxi -z 11333=localhost:11311 -t 1 python t/moxi_mock.py

For vbucket development, start the following...

ruby ./t/rest_mock.rb

Then start a "pretend" memcached server...

./moxi -vvv -p 11311

Then...

./moxi -vvv -z url=http://localhost:4567/pools/default/bucketsStreaming/default -Z port_listen=11211

Then...

telnet localhost 11211

More notes:

If using Linux, you need a kernel with epoll (it's better than select()).

epoll isn't in Linux 2.4 yet, but there's a backport at:

http://www.xmailserver.org/linux-patches/nio-improve.html

You want the epoll-lt patch (level-triggered).

If you're using MacOS, you'll want libevent 1.1 or higher to deal with a kqueue bug.

Also, be warned that the -k (mlockall) option to memcached might be dangerous when using a large cache. Just make sure the memcached machines don't swap. memcached does non-blocking network I/O, but not disk. (it should never go to disk, or you've lost the whole point of it)

The memcached website is at:

http://www.danga.com/memcached/

Using moxi with libmemcached (instead of libvbucket)...

If you want to use moxi explicitly using ketama / consistent-hashing, you'll need to compile moxi with libmemcached.

First, you would configure moxi differently at build-time...

./configure --enable-moxi-vbucket=no
CFLAGS=<libmemcached/include>
LDFLAGS=<libmemcahed/lib>

After building, moxi understands the following kind of command-line...

moxi -z <port=>

moxi -z <port=<memcached_host:memcached_port(,*)>>

moxi will listen on the given port and accept connections from upstream memcached clients. It will forward requests to downstream memcached servers running on memcached_host:memcached_port. For example...

moxi -z 11211=my_memcached_server:11222

Above, moxi will listen on port 11211 and forward requests to the memcached running on my_memcached_server that's listening on port 11222.

You can list more than one memcached_host:memcached_port, separated by commas. For example...

moxi -z 11211=memcached_1:11211,memcached_2:11211

The default downstream port is 11211, so you can also just use...

moxi -z 11211=memcached_1,memcached_2

If you have some memcached servers running on the same server, but on different ports, you can put moxi in front of them with something like...

moxi -z 11211=server:11222,server:11233