Stream-Framework
Stream-Framework copied to clipboard
tschellenbach
Feature: Configurable backends for realtime functionality
Any ideas to plug in Mongodb or mysql as disk backed store instead of Cassandra.
PS: I read in documentation "Stream Framework uses celery and Redis/Cassandra to build a system with heavy writes and extremely light reads." -- Suggestion to strengthen the read(s) capability of the framework.
This ticket is about the realtime functionality (listening to changes to the newsfeed)
As for the main storage backend we currently support Redis & Cassandra. We are considering adding a database backend. The DB backend won't work well for large production deployments, but would give you an easy way to get started.
For small deployment I currently recommend starting with Redis as it's easier to manage compared to Cassandra.
@tschellenbach thanks for your valuable feedback.
"For smaller deployment ..." what are the upper limit(s) for the framework / size it can handle, no of users & redis memory.
this will give us insight on timeline to build an alternative or tweak the code.
<1m users the most common choice is Redis After that it depends on your feature set, but eventually most apps switch to Cassandra due to the lower costs and easier growth (no manual resharding).
we are working on "push on change" system like twitter / pinterest rather than facebook's apprach of "pull on demand".
We are thinking to Use redis for faster reads (temp db) and mongo / mysql / cassandra for permanent storage which can be use to populate redis.
PS: can i ping you on ur email to get feedback for future.
Of course, feel free to get in touch.
PS. getstream.io might also be a good option for you guys
I have a use-case where I want to use sharded Redis pub/sub channels to send out ephemeral real-time events/messages (not stored in the DB), so that Node.js further in the stack can pick them up and send them out to iOS/Android apps. Also, to not flood the Redis server, when i.e. a post is created, I don't fanout 1kk messages per subscriber, but send 1 message per shard and let Node.js figure out what to do with them.
Similar to this approach http://code.flickr.net/2012/12/12/highly-available-real-time-notifications/.
Question: Is this framework applicable for such a use case?
yes you can build something very similar to what the article shows. My advise is to not do what the article suggest to do. If you are using stream-framework you already have a task broker (RabbitMQ) and background workers (Celery workers). This will work much better than the spaghetti infrastructure from the linked article and it will not introduce anything new in your infrastructure :) The main "limitation" would be that push to iOS/Android would be using Python instead of NodeJS. (but that does not sound that bad since you are using a python framework already) I hope this makes sense and that helps you, if not feel free to reply here or send me an email on [email protected] ;)
