Any support for RabbitMQ ?

[Niloy-Chakraborty]

I have RabbitMQ and I need to feed the streaming data for parallel execution. So,I opted RabbitMQ-> Streamz -> Dask for this purpose. But I am facing problems for Rabbit-Streamz integration. Any suggestions/solutions for this ?

[CJ-Wright]

What problems are you facing? The approach I took with ZMQ was to use a dispatcher which called source.emit when new data was sent over.

[Niloy-Chakraborty]

I am newly applying the concepts of Message Queue and Streaming for my work. So I am not getting how to consume the data using Streamz, from RabbitMQ

[Niloy-Chakraborty]

@CJ-Wright , could you please explain little more, if possible through code...It would be of great help.

[CJ-Wright]

Here is a short sketch.

1. Have a system which receives data and calls a function when that data is obtained, see here
2. Build a pipeline see streamz docs on how to do this.
3. When the data comes in from the message bus have the dispatcher (see 1) send the data into the pipeline.

An implementation of this lives here which is a bit much because it handles its own data model, but is roughly there, we define a dispatcher rd and then subscribe a callable to it rd.subscribe(rr) which pushes data into the pipeline.