portals icon indicating copy to clipboard operation
portals copied to clipboard

Dataparallel, distributed execution

Open jspenger opened this issue 1 year ago • 0 comments

There should be a distributed execution mode for Portals. The workloads should be split into partitions, and executed across distributed nodes. The distributed execution should provide fault-tolerance. The nodes are to communicate via Kafka (or similar) queues. Ask @jonasspenger for more details.

jspenger avatar May 09 '23 10:05 jspenger