Feature request: Namespace support

[mjrusso]

trafficstars

Proposal:

r = redis.Redis(namespace='foobar')

Behind-the-scenes, this would prefix every key with foobar.

For example, if I issued the following command:

r.set('abc', 'xyz')

It would be equivalent to the following:

redis-cli set foobar:abc xyz

In Ruby, the redis-namespace class does the same thing.

However, I think that it would be preferable if this were a core feature of the Python client.

[andymccurdy]

This seems like a reasonable suggestion. I'll add it in the next version. Remember that the namespace string will add extra bytes to each key. It might be better in some cases to namespace with the db=X option, since it has no additional memory overhead.

[kmerenkov]

What is the use case in real life (development or production)?

[mjrusso]

This allows the user to run several "applications" on top of a single Redis install in a manner that is more flexible than using different databases and different Redis processes.

This feature becomes increasingly useful as the number of utilities built on top of Redis increases, because namespace support allows multiple apps to be run against a single Redis install without fear of keys overlapping.

For example, Resque ( http://github.com/defunkt/resque ), uses namespacing to enable users to run multiple Resque instances in a straightforward manner. (More details are in the Namespaces section of the README.) The feature also lets this same Redis instance be used for a different purpose, in addition to Resque.

It's most useful in a development context though. Right now I'm using Redis for over a dozen different purposes and I need to do this constant juggling act on my main dev machine (and, in general, be very careful that keys don't collide).

[kmerenkov]

Okay so I came up with an ugly but obvious solution. http://github.com/kmerenkov/redis-py/tree/namespaces

I use a decorator and decorate nearly all commands.

Pros: obvious Cons: uncool :-)

[andymccurdy]

@kmerenkov: Check out the consistent_hashing branch in my redis-py repo. I needed a way to detect which arguments in a command were keys just like you do, but did it a bit differently, similar to the way response callbacks work. I was envisioning using that for namespacing as well, barring a more elegant solution.

[dottedmag]

I feel I probably too late to suggest it, but such namespacing is easily done as a separate wrapper. Crude attempt is available at http://gist.github.com/616851 - it needs more special-casing for functions which don't follow (key, *args) signature, but it works pretty well for me.

[nikegp]

Hey Andy, any updates on this feature? It's really useful for development when you have many developers working on the same instance. Especially if you have many redis instances in your topology

Thanks

[binarymatt]

I started down the road of doing a wrapper that adds namespacing on top of redis-py. Here are the beginnings of what I was thinking: https://gist.github.com/2190617. It's not complete, but if there is still interest in adding ns to redis-py, i'd love to get feedback and possibly switch this over to something like a pull request.

[zmsmith]

I started playing around with an implementation here:

https://github.com/zmsmith/predis/blob/master/predis/client.py

It's based on StrictRedis using consistently named parameters, but it could definitely be changed to be more configurable. Also, it doesn't alter any calls that return keys right now, but it's a start.

[grimnight]

Any progress on this? It would be nice to have namespaces.

[softwaredoug]

I have a Python project, subredis, that implements this nicely. A Subredis is a nearly full StrictRedis implementation/wrapper that uses a namespace. Its definitely a useful pattern. Some thoughts on limitations:

• Lua scripting becomes challenging. How do you replace references to keys in Lua scripts with a namespaced key? In subredis I don't allow this.
• How much do you sandbox the namespace? Should the namespaced redis be able to do admin-like commands/config like bgsave, etc? In subredis, I don't allow this.

[paicha]

2010-2015

[naktinis]

@andymccurdy You've mentioned you could "add it in the next version". Is it still an option? I believe it would be a welcomed feature. Especially now that multiple databases are discouraged.

Edit: multiple databases might not be deprecated for non-cluster mode after all.

[Kudo]

+1

[joshowen]

+1

[thestick613]

+1?

[exit99]

@andymccurdy Here is a working version with tests. I can expand the testing to all commands if you like the direction. https://github.com/Kazanz/redis-py/commit/8994f2633e60294573cdbeac452802ed1002d24a

[andymccurdy]

@kazanz your solution only works for commands that deal with exactly one key. Commands like KEYS, SUNION/SINTER, SORT, and others that operate on 2 or more keys would not be covered. Also, any command that doesn't operate on a key, e.g. INFO, CONFIG etc. would also break with this change. It's one of the reasons why this issue is tricky and hasn't already been addressed.

[exit99]

@andymccurdy I see. If no one else is addressing the issue, I would be happy to. Let me know.

[exit99]

@andymccurdy Here is a working version that allows namespacing on every command along with tests for every command. Works in py2 & py3. Followed proposed syntax of @mjrusso. Pull request here.

[etcsayeghr]

6 years later, still no namespace?

[andymccurdy]

@etcsayeghr Pull requests are accepted. This is a pretty complicated issue that I personally don't have a need for.

You can see some of the complexities here: https://github.com/andymccurdy/redis-py/pull/710#issuecomment-174850384

[exit99]

@andymccurdy Finally have some more time to refactor based on your comments. Will post periodically for feedback.

[exit99]

Finished a working PR with namespace support for all cmd calls on:

• StrictRedis
• Redis
• PubSub
• Pipeline
• Lua Scripting (only for keys passed to the script)

Refactored based on @andymccurdy comments on my last PR.

New PR uses simple decorators to handle namespacing. Followed proposed syntax by @mjrusso.
Has modified readme with usage instructions. Be sure to look at readme as some previous code may need changing if you are applying a namespace.

Includes full test suit for python2 + python3.

Here is my response to the complexities mentioned.

Thanks

[exit99]

@andymccurdy Any update on this pull request? We use this in production at my company and would like to see it integrated so we don't have to maintain a separate installation.

[andymccurdy]

@Kazanz Hey, sorry I've been away visiting family and friends for the past 8 weeks. When I last looked at this, there were some commands that weren't implemented (SORT comes to mind) and some commands that looking at the code didn't seem like they'd work correctly (for example ZADD has multi=True, but that won't work if the score values are strings (the scores would get prefixed) and it also wouldn't work if someone passed name/score pairs as **kwargs).

After discovering a few of these types of issues I went off into the weeds and made a separate implementation. I made execute_command take a parameter called keys_at. The value can be one of 3 things:

• an int that is the index of a single key in the args. commands like GET use keys_at=1, where the args are ('GET', 'my-key').
• a list of ints. same idea as above, but can specify multiple keys. great for commands like SORT where there's no defined pattern to where keys occur in the command string.
• a slice created with slice(). commands like MGET use keys_at=slice(1, None), and MSET uses keys_at=slice(1, None, 2).

keys_at tells us then specifically which args in a command string are keys so that they can be prefixed.

Although still incomplete (I haven't dealt with commands that return values like KEYS), I think it's much more straight forward and easier to understand what's happening. I'll make a branch and push my work.

[exit99]

@andymccurdy Having 8 weeks for family and friends must be nice :).

I like the keys_at parameter, would definitely be more readable.

Perhaps we could also have a keys_returned_at parameter for functions that could return keys?

When you get your branch up, I will go through and write defaults for each command and anything else you think needs to be addressed.

Happy to be moving forward on this.

[manusha1980]

@andymccurdy May I know the latest status of this please?

[exit99]

@andymccurdy Any chance your going to be pushing that branch soon?

Again, I am happy to go through and define the keys_at parameter for each command and anything else you think needs to be addressed.

If you havn't got a chance to write the code, I'm happy to write something along the lines of your previous post.

Let me know.

[AngusP]

Adding to this, it may make sense also supporting hash tags for those running a cluster of Redis servers https://redis.io/topics/cluster-spec#keys-hash-tags