BatchFlow icon indicating copy to clipboard operation
BatchFlow copied to clipboard

Make processing distributed

Open Teun opened this issue 13 years ago • 0 comments

It would be really nice if BatchFlow could leverage processing power from multiple machines. There are a number of hard nuts to crack here.

#. I think that Redis would be very suitable for the central storage of the queue content. It supports distributed atomic storage and blocking POP calls on queues #. Some of the patterns that are currently very common for use in BatchFlow will not be (easily) possible in distributed mode. State stored in the closure of lambda expressions will be problematic. Actually, task nodes that are initialized in the current way (using a lambda in the contructor) will not be possible anyway, as they cannot be serialized and deserialized on another machine

TBC

Teun avatar Jul 18 '12 13:07 Teun