redis-shatter

redis-shatter copied to clipboard

redis-shatter

redis-shatter is a sharding proxy for the Redis protocol, inspired by twemproxy (https://github.com/twitter/twemproxy). Documentation on how to actually use this is in the example configuration file, redis-shatter.conf.

In essence, this proxy appears like a standard Redis server to other hosts, but doesn't store any data locally - instead, keys are distributed between multiple backends (other hosts that speak the Redis protocol, usually Redis servers). This can give useful speed, reliability, and scalability benefits, but has some significant drawbacks as well:

• Command execution requires another network hop, which comes with a performance penalty. Clients that pipeline commands are somewhat insulated from this issue.
• Complex commands that affect multiple keys cannot be run efficiently unless the affected keys are all on the same backend. There are inefficient ways to run these commands, but these are (currently) not implemented by redis-shatter.

Performance

redis-shatter in front of 8 Redis server instances on the same machine was about 60% as fast as a single Redis server itself, according to side-by-side redis-benchmark tests. Very little performance optimization work has been done, so there's probably a lot of room for improvement here.

Like most of my projects, this is only tested at a small scale (so far), so there may be unfound bugs or inefficiencies. Use at your own risk.

Behavior

Unlike other similar projects, redis-shatter strives to implement as many commands as possible. For some of the less-common commands, redis-shatter deviates from the standard Redis protocol in order to implement reasonable behavior. These deviations are explained in the notes in the below table.

Command -- Supported -- Notes (see end)

ACL CAT -- Yes -- *E ACL DELUSER -- Yes -- *6 ACL GENPASS -- Yes -- *E ACL GETUSER -- Yes -- *5 ACL HELP -- Yes -- *E ACL LIST -- Yes -- *5 ACL LOAD -- Yes -- *6 ACL LOG -- Yes -- *5 ACL SAVE -- Yes -- *6 ACL SETUSER -- Yes -- *6 ACL USERS -- Yes -- *5 ACL WHOAMI -- No -- APPEND -- Yes -- AUTH -- No -- BGREWRITEAOF -- Yes -- *6 BGSAVE -- Yes -- *6 BITCOUNT -- Yes -- BITFIELD -- Yes -- BITOP -- Yes -- *2 BITPOS -- Yes -- BLPOP -- No -- BRPOP -- No -- BRPOPLPUSH -- No -- BZPOPMAX -- No -- BZPOPMIN -- No -- CLIENT CACHING -- No -- CLIENT GETNAME -- Yes -- *F CLIENT GETREDIR -- No -- CLIENT ID -- No -- CLIENT KILL -- No -- CLIENT LIST -- Yes -- *C CLIENT PAUSE -- No -- CLIENT REPLY -- No -- CLIENT SETNAME -- Yes -- *F CLIENT TRACKING -- No -- CLIENT UNBLOCK -- No -- CLUSTER -- No -- *H COMMAND -- Yes -- *E COMMAND COUNT -- Yes -- *E COMMAND GETKEYS -- Yes -- *E COMMAND INFO -- Yes -- *E CONFIG GET -- Yes -- *5 CONFIG RESETSTAT -- Yes -- *6 CONFIG REWRITE -- Yes -- *6 CONFIG SET -- Yes -- *6 DBSIZE -- Yes -- *A DEBUG OBJECT -- Yes -- DEBUG SEGFAULT -- No -- DECR -- Yes -- DECRBY -- Yes -- DEL -- Yes -- *4 DISCARD -- No -- DUMP -- Yes -- ECHO -- Yes -- EVAL -- Yes -- *0 *2 EVALSHA -- Yes -- *0 *2 *G EXEC -- No -- EXISTS -- Yes -- *4 EXPIRE -- Yes -- EXPIREAT -- Yes -- FLUSHALL -- Yes -- *6 FLUSHDB -- Yes -- *6 *7 GEOADD -- Yes -- GEODIST -- Yes -- GEOHASH -- Yes -- GEOPOS -- Yes -- GEORADIUS -- Yes -- *2 GEORADIUSBYMEMBER -- Yes -- *2 GET -- Yes -- GETBIT -- Yes -- GETRANGE -- Yes -- GETSET -- Yes -- HDEL -- Yes -- HELLO -- No -- HEXISTS -- Yes -- HGET -- Yes -- HGETALL -- Yes -- HINCRBY -- Yes -- HINCRBYFLOAT -- Yes -- HKEYS -- Yes -- HLEN -- Yes -- HMGET -- Yes -- HMSET -- Yes -- HSCAN -- Yes -- HSET -- Yes -- HSETNX -- Yes -- HSTRLEN -- Yes -- HVALS -- Yes -- INCR -- Yes -- INCRBY -- Yes -- INCRBYFLOAT -- Yes -- INFO -- Yes -- *8 KEYS -- Yes -- *9 *A LASTSAVE -- Yes -- *5 LATENCY DOCTOR -- Yes -- *5 LATENCY GRAPH -- Yes -- *5 LATENCY RESET -- Yes -- *5 LATENCY LATEST -- Yes -- *5 LATENCY HISTORY -- Yes -- *5 LATENCY HELP -- Yes -- *E LINDEX -- Yes -- LINSERT -- Yes -- LLEN -- Yes -- LOLWUT -- Yes -- *E LPOP -- Yes -- LPUSH -- Yes -- LPUSHX -- Yes -- LRANGE -- Yes -- LREM -- Yes -- LSET -- Yes -- LTRIM -- Yes -- MEMORY DOCTOR -- Yes -- *5 MEMORY HELP -- Yes -- *E MEMORY MALLOC-STATS -- Yes -- *5 MEMORY PURGE -- Yes -- *5 MEMORY STATS -- Yes -- *5 MEMORY USAGE -- Yes -- MGET -- Yes -- *4 MIGRATE -- Yes -- *4 *J MODULE LIST -- Yes -- *5 MODULE LOAD -- Yes -- *5 MODULE UNLOAD -- Yes -- *5 MONITOR -- No -- MOVE -- No -- MSET -- Yes -- *4 MSETNX -- Yes -- *2 MULTI -- No -- OBJECT ENCODING -- Yes -- OBJECT FREQ -- Yes -- OBJECT IDLETIME -- Yes -- OBJECT REFCOUNT -- Yes -- OBJECT HELP -- Yes -- *E PERSIST -- Yes -- PEXPIRE -- Yes -- PEXPIREAT -- Yes -- PFADD -- Yes -- PFCOUNT -- Yes -- *2 PFMERGE -- Yes -- *2 PING -- Yes -- PSETEX -- Yes -- PSUBSCRIBE -- No -- PTTL -- Yes -- PUBLISH -- No -- PUBSUB -- No -- PUNSUBSCRIBE -- No -- QUIT -- Yes -- RANDOMKEY -- Yes -- *1 READONLY -- No -- READWRITE -- No -- RENAME -- Yes -- *2 RENAMENX -- Yes -- *2 REPLICAOF -- No -- *H RESTORE -- Yes -- ROLE -- Yes -- *K RPOP -- Yes -- RPOPLPUSH -- Yes -- *2 RPUSH -- Yes -- RPUSHX -- Yes -- SADD -- Yes -- SAVE -- Yes -- *6 SCAN -- Yes -- *A *B SCARD -- Yes -- SCRIPT DEBUG -- No -- SCRIPT EXISTS -- Yes -- *D SCRIPT FLUSH -- Yes -- *6 SCRIPT KILL -- No -- *H SCRIPT LOAD -- Yes -- *6 SDIFF -- Yes -- *2 SDIFFSTORE -- Yes -- *2 SELECT -- No -- SET -- Yes -- SETBIT -- Yes -- SETEX -- Yes -- SETNX -- Yes -- SETRANGE -- Yes -- SHUTDOWN -- Yes -- *6 SINTER -- Yes -- *2 SINTERSTORE -- Yes -- *2 SISMEMBER -- Yes -- STRALGO -- No -- SLAVEOF -- No -- *H SLOWLOG -- Yes -- *5 SMEMBERS -- Yes -- SMOVE -- Yes -- *2 SORT -- Yes -- *2 *3 SPOP -- Yes -- SRANDMEMBER -- Yes -- SREM -- Yes -- SSCAN -- Yes -- STRLEN -- Yes -- SUBSCRIBE -- No -- SUNION -- Yes -- *2 SUNIONSTORE -- Yes -- *2 SWAPDB -- No -- SYNC -- No -- TIME -- Yes -- *6 TOUCH -- Yes -- TTL -- Yes -- TYPE -- Yes -- UNLINK -- Yes -- *4 UNSUBSCRIBE -- No -- UNWATCH -- No -- WAIT -- No -- WATCH -- No -- XACK -- Yes -- XADD -- Yes -- XCLAIM -- Yes -- XDEL -- Yes -- XGROUP -- Yes -- XINFO -- Yes -- XLEN -- Yes -- XPENDING -- Yes -- XRANGE -- Yes -- XREAD -- Yes -- *L XREADGROUP -- Yes -- *L XREVRANGE -- Yes -- XTRIM -- Yes -- ZADD -- Yes -- ZCARD -- Yes -- ZCOUNT -- Yes -- ZINCRBY -- Yes -- ZINTERSTORE -- Yes -- *2 ZLEXCOUNT -- Yes -- ZPOPMAX -- Yes -- ZPOPMIN -- Yes -- ZRANGE -- Yes -- ZRANGEBYLEX -- Yes -- ZRANGEBYSCORE -- Yes -- ZRANK -- Yes -- ZREM -- Yes -- ZREMRANGEBYLEX -- Yes -- ZREMRANGEBYRANK -- Yes -- ZREMRANGEBYSCORE -- Yes -- ZREVRANGE -- Yes -- ZREVRANGEBYLEX -- Yes -- ZREVRANGEBYSCORE -- Yes -- ZREVRANK -- Yes -- ZSCAN -- Yes -- ZSCORE -- Yes -- ZUNIONSTORE -- Yes -- *2

Notes: *0 -- Scripts that affect no keys will run on a random backend. *1 -- Distribution of random keys may not be exactly uniform. RANDOMKEY is implemented by choosing a random backend and sending RANDOMKEY to it, so if backend A has more keys than backend B, the probability of returning each key from backend B is higher. *2 -- The affected keys must all be on the same backend. If they aren't, the command fails with PROXYERROR. *3 -- The proxy does not check that all the affected keys are on the same backend; the application has to do this itself. (This is because complex patterns may be given in e.g. the GET clause.) *4 -- These commands are atomic only on each backend; they are not atomic across all backends. *5 -- The proxy's response format is different than that of Redis - the proxy returns a multi response with one field per backend. If this isn't what you want, you can use FORWARD to interact with a single backend at once. *6 -- These commands are forwarded to all backends. *7 -- Since the proxy doesn't support multiple redis DBs, FLUSHDB is essentially equivalent to FLUSHALL. *8 -- INFO syntax is different from what redis-server expects. These are the valid forms of the INFO command in redis-shatter: - INFO - return proxy information - INFO BACKEND - return proxy stats for a backend - INFO [section] - send INFO to a specific backend and return its response verbatim For the first two forms, the response format is different as well, but should be mostly self-explanatory. The third form is equivalent to the command FORWARD INFO [section]. *9 -- May be slow and consume lots of memory if run on large datasets. It's almost always better to use SCAN instead. *A -- Keys may be considered or returned multiple times or be inaccessible through the proxy if they exist on the wrong backend. For example, if 'x' belongs on backend 3 but exists on backends 5 and 3, then 'x' will appear twice in the KEYS or SCAN results, or will be double-counted when running DBSIZE. If 'x' exists on backend 5 and not 3, then 'x' will appear once in the KEYS/SCAN results but 'GET x' will return nil. *B -- SCAN over the entire keyspace is implemented by scanning on each backend in turn. The proxy uses the highest-order bits of the cursor to keep track of which backend is currently being scanned. If the backend returns a cursor that has any of these bits set, the scan will fail. Most practical setups shouldn't run into this limit. *C -- The returned fields are different from redis-server. They are: - name: the client's name (this includes the host:port string). - cmdrecv: number of commands received from this client. - rspsent: number of responses sent to this client. - rspchain: response chain length (the number of responses that haven't been sent to the client yet because there isn't enough information to generate the response - some backends haven't replied yet). *D -- 1 will be returned for a script only if it exists on all backends - if it is missing on one or more backends, 0 is returned. *E -- These commands are implemented by forwarding them to a random backend. If the backends are not configured identically (i.e. some have rename-command directives in their configs and some don't) then the results may differ between subsequent calls. *F -- Names pertain only to the connection between the client and the proxy. *G -- EVALSHA is more likely to fail in a sharded environment, since the script needs to be loaded into all the backends' script caches for it to work on arbitrary keys. Although EVAL implicitly loads the script into the script cache, EVAL is only forwarded to one backend at a time. If you plan to run a script many times on different keys, use SCRIPT LOAD first to load the script on all backends at once. *H -- Note that any unsupported commands can still be run on individual backends by using the FORWARD command, but be careful when doing this. See below. *I -- Which command referred you to this note? *J -- If any backend returns an error, then MIGRATE returns a multi response containing the responses from all backends. If no backend returns an error, then MIGRATE returns NOKEY if all backends returned NOKEY, and OK otherwise. *K -- This command returns a multi response with two fields. The first field is the string "proxy"; the second field is a multi response containing the names of all of the backends. *L -- Blocking reads are not supported (the BLOCK argument to these commands must not be given).

Administration

redis-shatter implements many administrative commands that are omitted in other similar proxies. See the commands table above for details.

redis-shatter also implements some administrative commands that aren't part of the official Redis protocol. These commands are:

BACKEND key [key ...] Returns the name of the backend on which the given key belongs, as a data response. If multiple keys are given, returns a multi response with one data element per key.

BACKENDNUM key [key ...] Returns the number of the backend on which the given key belongs, as an integer response. If multiple keys are given, returns a multi response with one integer element per key.

BACKENDS Returns a list of all backends. The items are formatted as "host:port@name".

FORWARD backend-name command [args...] Forwards the given command directly to the given backend and return its response verbatim. You can use this in lieu of connecting directly to the backend. This command shares the backend connections with all other clients, so forwarding commands that affect connection state like MULTI or SELECT will cause misbehavior.

FORWARD "" command [args...] Forwards the given command to all backends and returns a multi response containing the backends' responses. As noted above, don't forward commands that affect connection state.

PRINTSTATE Prints the proxy's internal state to stderr.